NIFFRA Image to text annotations
Image
Bounding BoxDuring my time at a logistics firm I was tasked with running a massive data annotation initiative for our computer vision team. The goal was to build a dataset that would help a multi-modal model better understand street level environments and warehouse spaces. We needed to move away from simple one word tags and instead focus on creating highly detailed descriptions of what was happening in each frame. I managed the pipeline that took raw footage from our delivery vehicles and turned it into structured text data that our machine learning engineers could actually use for training. The actual labeling tasks were quite demanding for our annotation teams. They could not just list the objects in a photo. They had to write complete sentences explaining exactly what an object was doing and where it was located. For instance they had to specify if a pedestrian was stepping off a curb or just standing on the sidewalk. They also had to transcribe text from street signs and billboards while documenting environmental factors like heavy rain or low light conditions. It required a lot of focus because the descriptions had to be completely objective and free of any assumptions about what might happen next in the video frame. In terms of scale this was a massive undertaking that lasted about four months. We processed over one million unique images across several different operational environments. The bulk of the data came from daytime city traffic but we also included a large percentage of night shots and rare weather events to ensure the model could handle edge cases. By the end of the project we had generated over three million individual captions because we frequently had multiple people look at the same image to ensure we captured different angles of the scene. Maintaining quality across millions of rows of text data was our biggest challenge. We set up a system where every single image was reviewed by multiple people to check for consistency. If the descriptions varied too much the image was automatically flagged for a manual review by a senior team lead. We also created a baseline set of pre approved images and secretly mixed them into the daily queues to test the accuracy of our workers. This strict verification process kept our data clean and allowed us to hit our target of ninety eight percent accuracy before handing the final dataset over to the engineering team.