Porting Boardspace.net using Codenameone

Executive summary:

In two months of intense work, I ported a large java program to IOS and Android using codenameone, and maintained a shared code base with the starting version.   There were many speed bumps along the way, but the successful result did not entail any substantial rewrites or major bodies of new code.   After the initial effort to get it working, I ported a second (related) large program in about a month, spent another 4 months tuning, completing features, and adding touch screen friendly behaviors to both programs.

The Problem Space:

Boardspace.net uses a java client to allow users to play real time board games over the internet.  The core code that runs all of the games is packaged in a small number of jar files, about a megabyte in all, built from 160 source files containing 50,000 lines of Java.     The bulk of the porting effort is to retain the functionality of this core, so that about 75 individual games, consisting of a further 800 source files and 300,000 lines of Java could be used essentially unchanged.  The total amount of compiled code in the system (with 75 games) is about 4MB, plus a lot of graphics.   As a porting problem there are several strong constraints. The biggest problem is the sheer size of the code.  Any porting approach that would require a substantial rewrite is a non-starter.  Reliable TCP streams are required.  Http transactions are also used. There is heavy use of custom graphics, constructed and scaled on the fly. On the other side, non-constraints include: Swing, fancy UI widgets, or third party libraries, are not required.  Standard Java APIs from the AWT era are all that are needed. The games UIs are already laid out adaptively for varying screen size and geometry,  and the UI is already known to work with single-finger touch interfaces.  The physical parameters of tablets won't be a problem, except the physically small size on some devices. There is no requirement to use platform specific capability (such as GPS) or services (such as embedding Ads). In view of the above, codenameone looked pretty attractive; TCP sockets are supported,  the user interface API is similar to AWT, and it promised to provide IOS and Android ports without major rewrites or paradigm shifts.

lobby and games in progress

The Plan:

My overall plan was to build the codenameone version so it shared all except the list of imports at the top of each file with the current code, so that in the future, the codename1 and standard java branches would be substantially identical. core Java classes would have to be used "as is". Examples are Image,  Graphics, Font.  These are pervasive in all the UI code, and trying to replace or substantially alter them is not a good plan. Where codenameone functionality is missing, or substantially different from standard Java, create adapter classes to either provide the missing functionality or bridge between what codenameone provides and standard Java.  Examples: Color,  Component (the base class for all windows), and Event. Where adapting or using as-is is not possible, drop into platform-specific code which implements a new API that is neither java or codename1, but which does what is needed.  Examples; Rectangles and Font selection.  Here, it is necessary, edit the code to use the new API, but once edited it will be the same in all branches. Initially, set aside all but one game.  All of the games are substantially similar, and should be used pretty much as-is if the core files are ported according to plan. First get the whole body of the codename1 branch through the compiler.  Then get it working.  Then back port the codename1 version to standard java and get that working, iterate until both versions are working and substantially identical (except for "import" lines at the top of each file).  Isolate irreconcilable differences in a few files that are not intended to be identical.

typical source compare between standard java and codename1 versions

Phase 1: getting through the compiler

Starting with a duplicate branch of my source hierarchy as a codenameone project, one file at a time, eliminate compiler errors.  This is a tedious and difficult process, because nothing works yet, nothing can be tested, and every error requires a treasure hunt and an educated guess how best to resolve it.   These guesses were frequently wrong, creating a backtrack, undo and re-edit cycle.

• Core classes such as Image, Font, and Graphics and String, which are provided by the environment and which are mostly created by the system rather than by calling "new", are not going to be subclassed.  Mismatches in the APIs are handled by adding new static methods to the "garbage" class G which will be differently implemented in codenameone and standard Java.   This localizes all the knowledge about differences in class G.   Example: standard Java operations on images typically require an ImageObserver as one of the arguments, where otherwise similar Codenameone methods do not, so define G.getWidth(image,imageObserver) where the Codenameone implementation will ignore the ImageObserver.
  
  
• Other common classes, such as windows and widgets, have to be subclassed or encapsulated, and any missing methods provided in the new class.  Initially these contain dummy methods that will throw an error if called.  This strategy accumulated about 100 "bridge" classes that cover the differences between Codenameone and standard Java functionality.  Examples: bridge.Container extends com.codename1.UI.Container,  bridge.Date encapsulates java.util.date.   Whenever the shared functionality is missing or probably not identical, create a new "not implemented" method in the bridge class to be fixed later.
  
  
• Lots of little pieces of functionality from standard Java libraries are just not there, or are in different places.  Examples: standard java Math.pow is implemented by MathUtil.pow; Integer.bitCount is missing from Codename1.  Locating these missing pieces is a frustrating treasure hunt, but frequently its possible to cannibalize the open source java implementation.
  

Two of the most difficult cases in this process proved to be Rectangle and Event.   These aren't particularly classed as bugs or problems, only as illustration of the kind of issue you can expect to contend with.

Phase 2: getting the Codename1 (Android) version working.

Since this was a large body of legacy code, there were few unit tests or test beds in place; but once the whole project compiles, it's simple to insert scaffolding to test various pieces. I had flagged a few key pieces as critical, likely to be troublesome, and separable from the overall application.

The first of these was networking.  The framework requires HTTP connections to negotiate login, collect information from the server site, and report problems back to be logged.  The games themselves require a TCP stream to be opened, and a continuous, reliable, low bandwidth chitchat with the server.   HTTP connections, using a low level interface, turned out to work perfectly. TCP sockets immediately proved to be unreliable.   My framework was already paranoid about the reliability of the streams, so the problems showed up immediately, rather than escaping into the operational code.  This prompted my first venture into the Codename1 VM.  I was able to find and fix the TCP problems in a few days, and got the changes pushed back into the main development line with remarkably little hassle.

The second likely source of problems I had identified was image manipulation.  My code uses an unusual strategy to construct images with transparency for use at run time, and unless that graphics pipeline could be preserved, a potentially huge redesign might be required.   Specifically, my transparent images are constructed from pairs of ordinary jpg images, which requires efficient access to the original image RGB values, and the ability to construct new images from raw RGB values.   Fortunately this turned out to be close to a non-issue.  All the essential methods for loading, manipulating, scaling and displaying images existed and worked as advertised.

There were many more steps like these to get the whole project working, but the underlying fact that the code was all known to work (in the original) greatly smoothed the process.  Hundreds of bridge classes with "Error, not implemented" methods were resolved one at a time, with the original behavior as a model for what needed to be done.

Phase 3: getting the Codename1 (IOS) version working.

Theoretically, after getting the Android port working, the IOS port should have been a matter of submitting a new build. In practice it didn't work out that way.  The IOS branch of Codenameone is a much less mature product, and there were quite  a few problems getting an initial version running at all, and then getting it running reliably. 

Sometimes these required very painful debugging because the IOS version crashed with no useful information about why.  For example, I spent 3 days generating builds containing less and less code, until I finally discovered than an extra "static main" method, left over from testing, was causing the whole program to crash immediately.  This has since been fixed; the build process flags it as  a build error.

On another occasion, IOS runs starting throwing errors, complaining of internal inconsistencies, which turned out to be that the implementation of "Random.nextLong()" was generating bad values.  Random had definitely been working the day before, and by the time I diagnosed and documented it, Random() mysteriously started working again.  Some development work at Codenameone had broken something for a few days, and I was the victim.

Several ordinary Java constructions related to interfaces, or default methods in interfaces, either generated build errors, or built but created buggy code that crashed or misbehaved.  These were mostly fixed by restructuring my source code to avoid the forms that the IOS branch of codenameone doesn't handle.  Some of these have since been fixed. 

Several IOS "code generator" errors caused misbehavior at  build time, or (worse) at runtime.  Some have been diagnosed and fixed, others have been worked around. 

Surprises along the way.

There were a number of noteworthy surprises that required changes along the way.

• CPU speeds.   As far as I can determine, all tablets are wimps, optimized for graphics and not much else.  This caused almost all of my games' robot players to be too slow to use.  About the best cpu speed I've seen in any tablet or phone is 15% the speed of a typical desktop computer.   A zippy robot opponent taking 10 seconds per move on a desktop becomes unbearably slow 60 seconds per move on a tablet - and that's the best case.   I had to implement a new set of procedures for rating the cpu speeds and robot speeds, and to politely decline to let tablet users play against most of the robots.  It's a big disappointment.
  
  
• Lack of memory I thought that a tablet with 512M of ram would have no trouble running my games, but it's not so.  Both android and IOS are optimized to run lots of small processes, and have a strategy of randomly killing old processes to make way for new ones.   They're not real computers.  In the event, older tablets with 512M are on the edge.  1G is better.
  
  
• Not a real window system.  Android and IOS both expect one active full screen process. Period.  Even if the physical and pixel sizes are generous.  My games expected to allow you to open multiple windows containing multiple games and the lobby where games are set up.   After some initial experiments and arguments with the codename1 staff, I acquiesced and implemented a tabbed interface where all my independent windows became full screen tabs.   The down side of this for me is that the layouts of game windows were "user optimized" by re-sizing them to a pleasant size.   A lot of the new, fixed sizes were not particularly pleasing, and there are a lot  of different screen geometries.   Making the game layout code more flexible is still an ongoing process.
  
  
• Unexpected use of Introspection, Clone, and Serialize.  Although I "knew" I didn't make use of these features, guess what, I did.  Mostly not in any hard to replace way, but a surprising number of cases popped up.
  
  
• Lack of thread safety.   All of the codename1 window system is not thread safe, and there are a lot of threads floating around.   Effectively all drawing activity, even offscreen drawing, has to be shifted to the main EDT process.   Random manipulations of windows such as changing text messages, adding or removing windows, also has to be shifted to the EDT process.
  
  
• Flickering displays Android displays, but not IOS, had a persistent problem with flickering.  Users expect a rock-solid picture.  It turns out there are deep seated problems using dynamically created bitmaps to speed up frame times on Android.  I'm avoiding these with some ad-hoc methods, but it's not completely satisfactory.
  
  
• IOS project build times  While my android build times remained stable at 1-2 minutes, as I brought more games into the system my IOS build time grew exponentially.  I finally stopped to study the issue when my typical IOS build time exceeded 1 Hour.  I eventually discovered that the IOS build has a "kitchen sink" approach to header files, so if you're not careful, even trivial classes will be compiled with all the headers for all the classes in the entire project.   Several careful restructurings of the classes, especially to avoid making classes even theoretically visible to one another, reduced the build time to 20 minutes.
  

Missing and/or Mis-features

• Weak Hash Maps are mis-implemented as "discard on GC" maps. Just don't event try to use them in their intended way.

• Font size and scaling.   Codename1 has a poorly developed system for selecting fonts with size appropriate to the screen pixel density, which varies a lot. 
  
  
• Zip format which I use to archive bundles of games, is not supported.  I think zip format is now semi-supported in an external library.
  
  
• Class.forName is not implemented on IOS.  Not only is there no dynamic loading of classes (which I expected) but you have to have an explicit list of the classes you want to instantiate.
  

The Codenameone bug/fix cycle:

Continuing annoyances

• Some features which I consider to be absolutely essential are relegated to "premium" status.   The two worst cases are crash logging and participation in Apple's beta program for new releases.
  
  
• The build servers periodically become unstable, start generating build errors for no discernible reason.  This problem has somewhat abated.
  
  
• IOS build sizes are really big as well as slow to generate.  My complete Android build is about 22M.  The corresponding IOS build, in the app store, is 120M
  
  
• Silver spoon bug reports are required.  If you're able to produce a single file, 100% reproducible test case, and submit it in just the right way, codenameone staff will probably respond appropriately.  Anything less than that is likely to be waved off, argued away, or ignored.

• Generally speaking, Codenameone has shown little interest in making their platform to be more compatible with standard Java, or for that matter more like any of the native platforms they support.  Codenameone is its own thing, and in their view you should target it, rather than expect it to target you.
  

Conclusion

Google Play;  Boardspace.net and Tantrix.com ITunes Boardspace.net and Tantrix.com.