This week we have a guest – an OSINT practitioner, a security enthusiast and a visual photo examiner, all in one…
Hey Charles, to begin with maybe tell us a bit about your background – why and how you became interested in OSINT and visual photo ID?
I have a background in ops management within startups, where I picked up a few useful tech skills, but on reflection since childhood It’s fair to say I’ve always been an inquisitive type..
I got interested in what I know now as OSINT in 2012/2013 when attempting to cobble together some method of real time fraud detection system for an on demand service product with poor merchant security. I found an organised and determined bunch of fraudsters (mainly via the OPSEC failures of their associates). In this role I also used various detection methods to compile a large intelligence dossier to give to the police regarding a notorious series of murders in London.
My interest has grown over time and I feel it’s clear organisations will rely more and more on good OSINT, one way or another. At work OSINT search techniques have been very useful, for example, in finding out detailed information on competitors/trends/new entrants etc..
I started a new identity in August to develop my OSINT, and happened across DCuplink who posted up some interesting OSINT challenges of which I was lucky enough to solve a few
What is your favourite approach to geo-locating photos?
Contextual clues are my favourite, for example knowing the type of person @DCuplink appears as on Twitter, and his image history, I was pretty sure his mountain image was taken near a place where US military personnel may pass through.
I recognise this may be counterproductive, but is a startpoint if I don’t have others or if I get stuck, or if I need to speculate on an aspect of an image.
When I start on an image I always look for:
- Things I can recognise.
- Things I can Reverse Image Search.
- Things I can use to confirm/deny my guesses.
I also try and use advanced Google searches to link elements within an image.
I would definitely recommend anyone interested in locating places to have a go at using the OpenStreetMaps API via https://overpass-turbo.eu/
Let’s talk examples – the 3 photographs you identified on Twitter, how did you go about it?
Initially, I felt strongly this was in the UK from the look, and the road sign. I took note of the sign/wall location as a means of confirmation. I initially searched on Yandex, and looked at a few water towers of a different type and their manufacturer. I also tried to RIS the Grit bin (to narrow down geographic area/municipal area) without success.
I then cropped out some of the background and did a search via Bing. After scrolling 10-15 pics, I saw a water tower of the same type (without the crosshatch pattern seen on many others), this linked to an Urban exploration site: https://www.28dayslater.co.uk/threads/best-of-2014.93674/.
This looked like it, I then confirmed by ‘driving’ there on Google Maps.
I learned from this to always RIS an image in each search engine FIRST, and to try a cropped image if this fails.
I was astounded by another Twitter user who located the tower by looking at the street sign to confirm the image was from UK UK, then searched a compendium of water towers on Wikipedia.
With a little more experience now, I’d consider using OpenStreetMaps’ Overpass Turbo to run a query for ‘Water Tower’ limited to the United Kingdom, which returns <100 results, one of which is in Heyford.
This was a tricky one, and highlighted the importance/risk of collaboration and contextual clues.
A variety of people had been guessing without success, and we were getting clues from @DCuplink. I misinterpreted one reply to mean the image was not in Italy.
As a result, I used contextual hints from the overall pattern of images set by @DCuplink to look for places a US military member may pass through, guessing Moron air base in southern Spain, near a mountain range.
Later, DCuplink makes a comment “I suspect the reason some of the heavy hitters have not spoken up in this search is it would reveal they know the location from personal experience. It’s a nice place. Very peaceful.”
This told me 1) This was a location related to the US Military, 2) This may well be a rehab/recovery place.
From there, Google searches for US Military Rehab Italy would have given Aviano. Luckily, another user had stated this confidently by paying attention to a posted video and seeing a plane I missed.
With the area narrowed down and another context clue “Can slightly defeat image searching” I panned around on Google Earth Pro, and noticed a small area by a dam that was somewhat covered by a mountain range. Additionally, I saw in a dry area what looked to be the jetty in the image. From there I googled the town name, saw images/trip reviews, and confirmed Barcis by the flags and distinctive crosshatch guardrail on jetty.
My learning here is all about organisation, confirming suspicions and working collaboratively if possible.
This was fun, helped out again by my knowledge of the UK. This was also the image that made me think hard about how to improve the method used.
I noted the telephone number on the Estate Agent/Realtor board was an Area code for London and kept in mind (this turned out to be misleading)
After trying reverse image search on the whole image, and distinctive cropped parts (the clocktower) I took note of the stores, those I knew to be chains. After listing them, I researched that the one with fewest stores was a retro UK diner style restaurant called “Jenny’s”.
A dedicated image search for Jenny’s gave me an image which showed both the mock-Tudor style frontage, and also the ‘Thomas Cook’ travel agency adjacent. Driving around in Google Maps and finding the clocktower confirmed it.
Interestingly, I saw (after) that another user who solved this did the same thing, but painstakingly searched Santander, a national bank with many more branches..
After some thinking, I got the idea to make a program where I can define three or more places (eg; Santander) and use a mapping API to locate ALL of these, then do the same for other places I see in the image.
With the Latitude/Longitude, I can use SQL to CROSS JOIN these places ON a Euclidean distance calculation to capture, for example, all locations in a given area (Eg; UK) with a McDonald’s, KFC and Pizza Hut WHERE max distance between each is X meters. This would solve this image, plus potentially many more.
This is perhaps not the most efficient way, but using Open Street Maps is free and frictionless for someone using the program. In addition, combined searches; for example “A KFC, a Gas station and a Park” are also possible. Unfortunately, I am new to coding/Python, but am halfway there and should have a working version available to the OSINT community very soon.
In your opinion, what are the biggest difficulties that a visual photo investigator could face? And how could they overcome those, whatever they are?
I feel that an investigator has to walk a fine path between empathy and dissociation. The inherent misery and sadness behind a given image could easily creep up on a person, however well adjusted they are. Indeed, the sheer scale of the problem vs the capacity of an individual to solve the problem compounds this. I am mindful of the case of Facebook moderators experiencing PTSD.
People in this field will have to take great care not to ‘take their work home’ and should have good access to others within the field with similar roles and, most importantly, professional psychological support as a prerequisite as opposed to upon request or once experiencing issues.
Other difficulties could include access to technological resources, for example; where someone wishes to work with very large data sets and/or perform mathematical functions on this data without access to (for example) cloud CPU/Storage facilities. I fear I may encounter this issue myself with a tool I am developing. Organisations with access to such facilities may wish to donate access as a goodwill/philanthropic gesture.
Lastly, there are a number of enterprise level tools that quite rightly cost a lot of money. I feel that the creators of these might well consider a ‘closed beta’ for individuals who have proven contributions to the OSINT (for good) community to use for Non-Commercial use. Indeed, if the creators could access the usage stats of these users, a symbiotic relationship may be established.
How do you feel about online investigators (both amateur and professional ones) geo-locating war crimes and atrocities on Reddit or Twitter?
I feel that this is a very noble endeavour. My reservations are wholly with regard to the open platform of Reddit/Twitter. I feel that the potential downsides of this could outweigh the benefits and a closed, attributable environment is the proper forum for both amateurs and professionals to collaborate. I am open and hopeful to being wrong on this issue and OSINT with regard to war crimes/atrocities is not something I have a strong knowledge of (yet).
In an article a few weeks ago I discussed Europol’s Trace An Object campaign – what is your take on it?
I really enjoyed this article and learned a lot from VisciousNakedMoleRat. I especially appreciated the tips on image manipulation, and have looked into some of the tools/methods mentioned. I also wonder why no dates are ascribed to images. I was most impressed by their interpretation of subtle context clues to create nuanced Google searches.
Overall, their approach is far more artful than mine currently, and I am in awe of their dedication and craftsmanship. I have an innate wish to attempt to mentally shoehorn such creative analysis into a decision tree and automate/scale the process, but fear some/most/all of the ‘magic’ would be lost.
Is there perhaps any room for improvement, something you think could be done even better, more effectively?
Researching the Trace an Object campaign has really resonated with me, and I’d like to take a creative interpretation of this question to speak at length on my own thoughts on the campaign as a whole, and also highlight suggestions I feel could improve traction/results.
Overall, I think the concept is amazing! Progressive, scalable and brings awareness and inclusion to a topic almost all people care deeply about yet so few can actively participate against.
However, as a visitor and prospective participant on the site, I am disappointed.
When I began to look into a few images myself, I was unhappy to find one I’d done initial searching on had been identified on Reddit months ago. The work of these Redditiors was seriously impressive, especially with regards to the opportunities of collaborative identification, and I joined that community (of 60k users!) Honestly, this did dampen my spirits toward the Europol initiative. Assuming the subreddit is managed/viewed by Europol, there are opportunities to synergise.
I would also like to see
- A set schedule of new images added, ideally with a means of subscription/notification.
- Success stories/removal of Identified images, thus showing what a good tip looks like.
- A recent update on any aspect of this project, the last press release is two years old.
- Some form of collaboration possibility, perhaps as part of a private, managed forum.
The crux of this initiative is professionals at Europol, with access to cutting edge tools, are unable to locate given images, and they are crowdsourcing help as a ‘shot in the dark’ and the more shots you take, the more hits you get. This is a simple yet elegant approach.
However, with 2,400 tips per victim ID, this approach as implemented seems noisy/costly.
So what do you suggest?
It is sobering to know Europol has over 40 MILLION images of child sex abuse and growing, but it’s also good to know about the wonderful efforts of volunteers via the Bellingcat update.
I feel progress comes in two main areas; 1) an improvement of detection probability per view and 2) an increase in views to facilitate an exponential increase in overall identifications via this campaign.
To improve detections per view, I would consider trying to harness the crowd to perform other tasks than just ‘SOLVE’
For example, ‘READ’, where users are asked if there is text in an image, and what character set/language it is. Another example might be ‘CATEGORY’ where a given image is intelligently sorted/categorized/tagged in a similar way to the ‘object’ setting in a game called Akinator https://en.akinator.com/game (which I assume could be trained/improved via pattern learning)
In this way, images can be defined to an extent, then segmented and served to users predicted to be most capable of solving (and this assumption can itself be checked and improved) This could work and be managed in a way very similar to personalised adverts and their conversion metrics.
Further to this, I would seek to VASTLY increase the number of impressions related to a given image, and take the audience beyond volunteers only.
For example:
- Partner with Ad-Block/replacement apps or create one to serve OSINT/VisualPhotoID tasks to users opting to replace ads with VPI, ideally filtered by user info and location image info as above. This method of replacing ads could be via DNS/DoH instead.
- Consider creating a CAPTCHA like service as a delivery method. By any definition, visual photo ID is a ‘hard’ task. Setting these “VPI” tasks as part of a Turing test to approximate a CAPTCHA service, and providing this as a standalone product for developers could be a very good way of attracting many more ‘hits’ from willing user participants. Each would be helping to progress a given investigation without encumbering human oversight capacity. With a little chutzpah, this could be marketed to web services most in need of intelligent DDOS/CAPTCHA protection.. (eg; EndGame) Indeed, a service such as a Darknet Market may be ideal for a variety of reasons..
- Lastly, I feel that the Trace An Object site could have resources to help aspiring developers within the OSINT space connect and collaborate. This would be helpful to improve the detection possibilities ‘upstream’ of the end user participation and could dramatically improve outcomes in terms of both successful identification, improving efficiency and being prepared for future challenges.
Awesome. Thank you kindly Charles. Is there anything else you’d like to add or discuss?
Thanks firstly to you Maciej for reaching out and DCuplink for setting cool and engaging challenges. Writing this up has been very interesting and a great learning experience for me. I hope to find a career in the OSINT or similar field and have found a wonderfully helpful and fascinating community.
If anyone would like to talk about OSINT, give me a hand coding an OSINT tool, or just say hello, I’m @CharlesWhiteCat on Twitter 🙂