Day summary

This day was all about the directory and password sync tools… We wanted to finalize their configuration to allow the support team the time for demonstrations and practices before going out to users and change their profiles…

We started out by working on GCDS, once we started working on the tool, we were faced with two critical problems:

1. Turned out the current number of licenses on the G Suite Admin Console is less than the actual number of users and less than what the customer has approved.
2. The OU that the customer wanted to sync was including disabled users, service accounts and a lot of user accounts with their ‘mail’ attribute value is wrong (it contained spaces in the beginning and end of the field).

We started out working on the second problem, since it is more visible to us and we knew what to do to fix it… The customer did not have an in-house Exchange Server, so there is a sync tool that syncs the user accounts with the Office 365. The email addresses attributes are being read form ‘proxyAddresses’ AD attribute, the capital case ‘SMTP:’ means that is the primary email address of the user, while the small case ‘smtp:’ means that is the alias(es) of the user; They only use the ‘mail’ attribute in AD for information purposes, so they did not care much about the accuracy of the data in it… That is a problem, because GCDS by default reads that value to create the users on Google…

The solution we had was to make a simulated sync, grab all accounts which the tool could not read their ‘mail’ attribute, and fixing that one by one.. we assigned the task to one of the IT team and we moved to the other larger problem…

We needed a way to control what gets synced to Google in a way we don’t mess things up for the support team… If we let it run and sync all what it can, there is a chance that a department does not get all its users synced, so needless confusion, user dissatisfaction, and other issues might arise… We can not wait for the remaining licenses to be activated because we were working in the end of year, and people are in their vacations and we had issues communicating with the people responsible for the licenses… so we needed to improvise and act.

We excluded 2 OUs from the AD, those 2 OUs had enough users to make the number of users to be synced below the limit of licenses available.

We ended up configuring the tool with explicit inclusion to all the OUs (and their sub-OUs) that we want to sync, and then we explicitly told the tool to only look for users inside those specific OUs… The point of this setup was while it is long and complicated, we have more control over what we can sync and not sync. It was meant to be a temporary setup until the licenses gets sorted out.

Based on the user lists we had the previous day, we had two types of licenses and users, business users and basic users. After we configured the tool and confirmed the setup through simulations, we did the first sync, and it worked greatly, however because we had 2 license types, one license got capped, and the tool stopped syncing users. Using GAM we assigned the business licenses to users and reran the sync tool, it added more users on the default basic license, and it got to full cap again, and again we used GAM to assign the proper licenses to users, and went on with this until we got all users sorted out.

We noticed that we had many users missing and after investigation we found out that those users among the users who are planned to be migrated, but they are on the list of excluded users… So we manually created them using GAM, and manually added exceptions to them on GCDS one by one.

This took most of the day, once we finished that last step we were ready with GCDS, I configured the scheduled task to run each hour and left it to do its thing. We moved then to the GSPS

Setting up and configuring the password sync tool was a piece of cake compared to what we went through earlier, in few minutes it was configured and running on all their AD servers. We called on the other team and demonstrated the password change process to them. Once they knew how it worked, they started using G Suite as core IT.

For some reason, not all the configuration that we did on GCDS were documented! This is going to cause a disaster later on, but no one will see that coming!

My notes and lessons learned

Working under pressure is something that we all should expect and see coming… however it is different feeling and experience from telling or hearing. When someone tells you that he can work under pressure or asks if you can work under pressure, the answer will be obvious that it is something can be done… However once you experience it, it turns to be something entirely different! Your mind needs to keep up with the constant flow of information, variables, data, and problems that comes on it, you have to process everything the moment it enters your system, it cannot be delayed or put on the to-do-list, if you leave something un-handled, you risk creating a seed of a future problem… This is exactly what happened with us on the point of documenting ALL the configuration that we did for the GCDS tool because we had pressure coming from outside the team (from end users, HR, management, some stakeholders who present a threat to the project if we delay certain things, etc…) we let an important aspect slip from us and went unnoticed even after we finished the tasks at hand… This is going to have its effects later on in the project and I can only say that I wish I had the required documentation correct.

I also learned that during times of emergency, we must not stop at any point, and everything has a solution or a workaround. If we stop the project wheel at some point because of an issue, it means we are incompetent and we cannot be entrusted to the integrity of the project in the coming days or weeks or months.

It is very and extremely important to find a solution to every problem, and to improvise to fix an issue or workaround a problem, while at the same time maintaining a record and a log for every step and configuration taken whether it is a legit, normal procedure or an emergent abnormal activity that was created or found to solve or fix a specific situation or condition.

It is is also very important to make sure any changes from the default or planned configuration or setup are made un-doable, there will be a time coming where you need to undo these non-default, emergency procedures or activities that you have taken to return back to the normal operation or status of the component/tool/item you have.

For me what I have done and learned in this day is greater than anything I had before, it put all my abilities and planning and mind to the test and I managed to get out with the perfect setup we can get.