How to Port a 9 Million Code Line Project to 64 bits?
- 1. How to Port a 9 Million Code Line Project to 64 bits? Authors: Ilya Ivanov, Evgeniy Ryzhkov Date: 03.08.2015 Our team has recently finished porting one pretty large project (9 million code lines, 300 Mbytes of source files) to the 64-bit platform. It took us one year and a half. Although we are not permitted by the NDA to disclose the project name, we still hope that our experience will help other developers in their work. About the authors We are known to many people as the authors of the PVS-Studio static code analyzer. Indeed, this is our major field of activity. But we also take part in various third-party projects as an expert team. We call it "expertise selling". Some time ago we published a report on accomplishment of our work on Unreal Engine 4. Today we present another report on the work we have done for our latest expertise selling task. "Aha, they must be having hard times with PVS-Studio then!", some readers following our activity may think. But we have to disappoint those eager for sensations. Participating in third-party projects is indeed a very important activity for our team, but for a different reason. It is the way for us to more actively use our own tool in real life, not only when working on its development. The everyday use of the analyzer in real-life commercial projects developed by dozens or even hundreds of developers is a source of great experience for the PVS-Studio team. We can see how people use our tool, what difficulties they face, and what we should change or improve in our product. That's why we are going to keep participating in such projects within the scope of our expertise selling activity. So feel free to contact us if you have a project we could help you with. And for now we are glad to present this report about code migration to the 64-bit platform.
- 2. Introduction, or what's the problem? Project terms and team size At first glance, the task of porting program code to the x64 platform looks clear and trivial. Back in 2010, we wrote our time-tested article "A Collection of Examples of 64-bit Errors in Real Programs". In 2012, we published our training course "Lessons on development of 64-bit C/C++ applications". All you have to do is follow the recommendations and guidelines given there and everything will be fine, right? Then why did the customer need to ask a third-party company (i.e. us) and why even we had to spend one year and a half to accomplish it? Since we implement analysis of 64-bit issues as part of PVS-Studio's functionality, we ought to know our way around the subject, oughtn't we? Yes, and we definitely know our subject very well, and it was the major reason why the customer contacted us. But why did they have to ask anyone for help with 64-bit migration of their code at all? Let's first say a few words about the project and the customer. Since we are not allowed by the NDA to reveal the information directly, I will only give you some figures. The project we were working on is about 20 years old. Presently there are a few dozens of programmers working on it every day. Their customers are large companies, and their sales are rare as their product is highly specialized. And the major characteristic is the code size. The source code includes 9 million code lines and has the overall size reaching 300 Mbytes, the solution (.sln) file embracing thousands of projects, which is a HUGE number. The target platform is Windows only. But even with that kind of a project, there shouldn't be any intricacies about the 64-bit migration task, should it? To port a project like that to the x64 platform, you only need to do the following: stop the development process completely for a few months; quickly replace data types with their 64-bit counterparts; make sure that everything runs well after the replacement; resume the development process. Why do we have "stop the development process completely for a few months" as the first step? Because you definitely need to replace some data types with 64-bit ones for successful 64-bit migration, of course. Creating a separate branch in a project of a size like that and doing all the necessary edits there just won't help because you will fail trying to merge it with the major code later! Keep in mind the project size and dozens of programmers adding new code every day. Due to some business driven limitations, the customer couldn't stop the development process. Their own customers would regularly need new releases, bug fixes, special features, etc. To stop the development process in circumstances like those would mean to stop the business entirely. That's why they set about seeking a team who could help to port the project without stopping the development process. They chose us because our competence in 64-bit development had been proved by our code analyzer PVS-Studio and articles on this subject. We accomplished this task in one year and a half. During the first half a year, there were two developers participating in the project on our part, and the next year there were four of us. Why did it take us so long? During the first half a year, those two developers were setting up the infrastructure, studying the project, and trying different migration algorithms. Then, as the task had become more well-defined, we added two more guys to the team and the project migration was finally accomplished by 4 programmers one year later. How to port a project to the 64-bit platform? On a large scale, the 64-bit migration task is comprised of the following two steps: 1. Creating a 64-bit configuration, obtaining 64-bit versions of third-party libraries, and project building.
- 3. 2. Fixing the code that causes errors in the 64-bit version. This step is almost completely reduced to the task of replacing 32-bit types with memsize-types. As you remember, memsize-types are those which have a variable size, in particular 4 bytes on the 32- bit system and 8 bytes on the 64-bit one. Porting a large-scale and intensively developing project shouldn't interfere with the current development process, so we made the following arrangements. First, we were doing all our edits in a separate branch so that we didn't break the major build. Once every new set of our changes was ready and tested, we would merge them with the trunk. Second, we decided to go without directly replacing 32-bit types with memsize-types. Instead, we created our own types to replace the original ones. It was done in order to avoid any potential troubles such as calling different implementations of an overloaded function, and also to leave an opportunity for us to rapidly roll back any changes if necessary. Our types were implemented in a pattern like this: #if defined(_M_IX86) typedef long MyLong; typedef unsigned long MyULong; #elif defined(_M_X64) typedef ptrdiff_t MyLong; typedef size_t MyULong; #else #error "Unsupported build platform" #endif I'd like to point out once again that we replaced the original types with our own ones, not with size_t/ptrdiff_t and the like. It gave us a lot of flexibility and allowed us to easily tell the already ported fragments from those yet "untouched by human hands". Various approaches to migration: Pros and Cons; our mistakes The first idea was the following: first we would replace all the 32-bit types with memsize-types except those fragments where 32-bit types had to be left untouched (for example structures implemented as data formats, and functions processing such structures) and then set it up to work. We chose this solution to eliminate as many 64-bit issues as possible at once and do it in one run, and then fix all the remaining warnings of the compiler and PVS-Studio. Although this approach works well for small projects, it didn't suit us this time. First, type replacement took too long and involved too many changes. Second, despite trying to be very careful, we still replaced a few structures with data formats by mistake. As a result, when we finished working on the first portion of the projects and ran the application, we failed to load the pre-installed interface templates as they were binary. So, the first plan implied the following algorithm. 1. Creating a 64-bit configuration. 2. Compilation. 3. Replacing most of 32-bit types with 64-bit ones (more exactly, with memsize-types). 4. Linking with third-party libraries. 5. Running the application. 6. Fixing the remaining compiler warnings.
- 4. 7. Fixing the remaining 64-bit issues detected by the PVS-Studio analyzer. This plan was recognized as unsuccessful. We accomplished the first five steps but had to roll back all the changes in the source code. That is, we had wasted several months. Now we decided to try getting a working 64-bit version of the application as soon as possible, and only then fix the most evident 64-bit issues. Our new plan didn't provide for mass type replacement and only implied fixing the most crucial 64-bit errors: 1. Creating a 64-bit configuration. 2. Compilation. 3. Linking with third-party libraries. 4. Running the application. 5. Fixing compiler warnings. 6. Fixing the most crucial 64-bit errors detected by the PVS-Studio analyzer. This time we got the first working version way sooner, also because we already had the third-party libraries built by the time and the interface templates loaded correctly. It should be noted that the application would run pretty stably most times, which surprised us very much. We had just a few crashes at the first testing. After that, we had to fix the compiler warnings and 64-bit warnings by the PVS-Studio analyzer to eliminate all the detected and potential crashes. Since the total number of 64-bit warnings by PVS- Studio reached thousands, we decided to only fix the most important ones: implicit conversions of memsize-types to 32-bit types (V103, V107, V110), conversions of pointers to 32-bit types and vice versa (V204, V205), suspicious conversion sequences (V220, V221), matching the types of virtual function parameters (V301), and replacing reciprocated functions with new versions (V303). See the documentation for descriptions of these diagnostics. In other words, the task at this stage was to fix all the 64-bit warnings of PVS-Studio under Level 1 only. These are the most crucial diagnostics, and all the corresponding issues must be fixed for a 64-bit application to launch successfully. Most of these edits in fact implied replacing 32-bit types with memsize-types, like in the first approach. But, unlike it, we did those replacements selectively and iteratively this time. It was determined by the fact that editing function parameter types led in its turn to editing the types of local variables and returned values, which, in their own turn, led further to editing the types of other functions' parameters, and so on, until the circled was closed. Another con of this approach as compared to the first one is that we could only fix the most critical 64- bit issues. For instance, counter types were left untouched: in most cases, we didn't need them, and they wouldn't cause any errors anyway. But in some fragments, I guess, we still should have got them fixed, and we had skipped them and wouldn't be able to find again if sticking to the chosen approach. In other words, we may have to do some additional fixes some time later. When porting the application, we also needed the 64-bit versions of the third-party libraries used in the project. For open-source libraries, we tried to build them from the same source files the 32-bit ones had been built from. The reason was that we wanted to keep all the possible bug fixes already done in their code, if any; and we also needed to build them, whenever possible, in the same configuration as for the 32-bit versions – for example with the compilation switch telling the compiler not to treat wchar_t as a built-in type or with a disabled Unicode support. In such cases, we had to play around with building parameters a bit before we could figure out why they wouldn't link with our project. Some libraries just weren't designed for re-building in the 64-bit version, in which cases we had to either convert them ourselves or download fresher versions allowing the 64-bit build. In case of commercial libraries, we
- 5. would either ask the customer to purchase the 64-bit version or look for a substitute for those of them which had no support any more, like it was with xaudio. We also had to get rid of all the assembler insertions as the 64-bit version of the Visual C++ compiler doesn't support Assembler. In this case, we would either use intrinsic functions wherever possible or rewrite the code in C++. In some cases, it didn't even cause any performance losses – for instance when the 32-bit assembler code fragments used 64-bit MMX registers, all the registers in our 64-bit version were already 64-bit. How long it takes to fix 64-bit errors in a project like that When only starting to work on a large-scale project, you can't say for sure how long its migration will take. We had to spend quite a lot of time at the first stage on building the third-party libraries, setting up the environment for daily builds of the 64-bit version, and testing. Once finished with the first portion of the projects, we could estimate our working speed based on the size of code already ported during a certain time period. Examples of 64-bit issues we were faced by The most common error when porting the project to the 64-bit platform appeared to be an explicit conversion of pointers to 32-bit types, for example DWORD. In such cases, we replaced those types with memsize-types. For example: MMRESULT m_tmScroll = timeSetEvent( GetScrollDelay(), TIMERRESOLUTION, TimerProc, (DWORD)this, TIME_CALLBACK_FUNCTION); We were also faced by errors when changing virtual function parameters in a base class. For example, the parameter type in CWnd::OnTimer(UINT_PTR nIDEvent) had changed from UINT to UINT_PTR when the 64-bit Windows version had been released, so we had to do this replacement in all the descendant classes in our project too. For example: class CConversionDlg : public CDialog { ... public: afx_msg void OnTimer(UINT nIDEvent); ... } Some WinAPI functions can work with large data amounts, for example CreateFileMapping and MapViewOfFile. So we modified the code accordingly: Before: sharedMemory_ = ::CreateFileMapping( INVALID_HANDLE_VALUE, // specify shared memory file pSecurityAttributes, //NULL, // security attributes PAGE_READWRITE, // sharing NULL, // high-order DWORD of the file size
- 6. sharedMemorySize, // low-order DWORD of the file size sharedMemoryName_.c_str()); After: #if defined(_M_IX86) DWORD sharedMemorySizeHigh = 0; DWORD sharedMemorySizeLow = sharedMemorySize; #elif defined(_M_X64) ULARGE_INTEGER converter; converter.QuadPart = sharedMemorySize; DWORD sharedMemorySizeHigh = converter.HighPart; DWORD sharedMemorySizeLow = converter.LowPart; #else #error "Unsuported build platform" #endif sharedMemory_ = ::CreateFileMapping( INVALID_HANDLE_VALUE, // specify shared memory file pSecurityAttributes, //NULL, // security attributes PAGE_READWRITE, // sharing sharedMemorySizeHigh, // high-order DWORD of the file size sharedMemorySizeLow, // low-order DWORD of the file size sharedMemoryName_.c_str()); We also found a number of functions which are considered reciprocated in the 64-bit version and must be replaced with the corresponding new implementations. For example, the GetWindowLong/SetWindowLong functions must be replaced with GetWindowLongPtr/SetWindowLongPtr. All the examples cited above as well as many other types of 64-bit issues can be found by the PVS-Studio analyzer. PVS-Studio's role in 64-bit migration Potential errors associated with 64-bit migration can be partially detected by the compiler. However, PVS-Studio is much better at this task since it has been originally designed to detect bugs of that kind. To find out more about what 64-bit errors PVS-Studio can detect which the compiler and Visual Studio's static analyzer cannot, see the article "64-Bit Code in 2015: New in the Diagnostics of Possible Issues" I'd like to mention one more important thing. Using the static analyzer regularly, we could see in real- time how old bugs were eliminated and some new 64-bit ones were added into the code. You see, the code is constantly being edited by dozens of programmers, and sometimes they make mistakes resulting in 64-bit errors in the project already adapted for x64 mode. But for static analysis, we wouldn't be able to tell for sure how many bugs had been fixed and how many had been added and how far we had
- 7. progressed. Thanks to PVS-Studio, we could draw diagrams to help us measure our progress. But this is another story. Conclusion To ensure as smooth 64-bit migration of your project as possible, you should stick to the following algorithm: 1. Study the theory (our articles, for example). 2. Find the corresponding 64-bit versions of all the libraries used in your project. 3. Build the 64-bit version as soon as possible and make sure it can compile and link well. 4. Fix all the 64-bit diagnostic messages of Level 1 (64 L1) by the PVS-Studio static analyzer. Additional reading on 64-bit migration 1. A Collection of Examples of 64-bit Errors in Real Programs. 2. Lessons on development of 64-bit C/C++ applications. 3. C++11 and 64-bit Issues 4. 64-bit Code in 2015: New in the Diagnostics of Possible Issues?