Human Error

Phillip Wulf’s Gravity Journal — Day 3

We are home now. Maria our golden retriever is curled at my feet as I update this journal. She loves staying with our grandkids, but they wear her out with walks, fetch, and dressup. She seems glad to be home.

I woke at 3:07 AM at the Columbia Point this morning to an email notification telling me my IGR credentials were ready. I couldn’t wait to try out the Grid. I was careful to set the alarm to vibrate so as not to wake Jianmin and to keep myself out of trouble. It only delayed the inevitable.

I allocated blocks of unused Grid bays in each of five locations including gravitational wave observatories at LIGO Hanford Washington, LIGO Livingston Louisiana, VIRGO Cascina Italy, GEO600 Hannover, Germany, and the radio telescope observatory at Arecibo, Puerto Rico. Most of the radio telescopes in the world now share observatory databases, so it was only necessary to connect one.

I did not want to wake Jianmin, so I issued written instructions to Varney, my AI assistant to scan project databases, and to run unit, regression, and design rule tests. I built Varney back in January 2018 using the Mycroft AI engine on GitHub. I had been following Mycroft for a while but did not fully embrace the program until Mycroft CTO Steve Penrod unveiled his autonomous vehicle efforts with Jaguar and Land Rover. I modeled Varney’s personality after British-American Stuart Varney, of Fox Business Network.

Varney had been running on a combination of Amazon AWS and custom servers I ran from my home office. I cobbled a new build script to let Varney run on the Grid. Varney could inspect the Grid without actually running on it, but I was curious to see the performance if he did. I issued instructions to Varney to recompile his own core with special debug traps should something go awry, giving him an option to proceed on the Grid, or return home to my lab and/or Amazon if he had to.

I had just enough time to peruse LIGO Transient Event Archives — the database where they store verified and candidate gravitational waves. I download copies of the Jupyter notebooks for all the confirmed GW captures and pending candidates to review later before Jianmin woke and made me close my MacBook — busted.

I dressed quickly and ran to get Starbucks coffee for both of us. She was dressed by the time I returned, and we had a lovely breakfast by a water feature inside the lodge great room. We cleaned up and packed, then paddled a tandem kayak around the Columbia Point marina for an hour, meeting and chatting with lots of nice people in boats.

Jianmin started her new IGR job while I took a nap. When I woke she told me she met with a dozen or so IGR project managers and various contributors around the world. She said there was a mountain of project status reports to analyze and an IT security project at our home she needed to manage. She had already contacted our architect and contractor to get things going. She said MI5 Mike and Steve would be helping us.

“Do you know what our titles are?” Jianmin asked?

“I don’t remember.” I said.

“You are Chief Data Scientist.”

“Okay.” I said.

“That’s a big deal, Phil. I am so proud of you.”

“What’s your title?” I asked.

“I am a Project Analyst 1.”

“I am proud of you too.” I said.

“You should be. I’m practically your boss.”

“You were already my boss.”

Jianmin has a doctorate in civil engineering from Tsinghua University in PRC. She and I worked together in China were we first met. This is her first job outside the home since we married and moved back to the US. She elected to be a stay-at-home-mom, chief domestic officer, and soccer mom with a PhD, and I am and will always be very proud of her.

The flight back home was a hoot. Steve the MI5 guy joined Jianmin and me along with Arni and Freya on our flight. Steve has a penchant for stand up comedy, and he entertained and charmed everyone aboard.

Varney called me during one of Steve’s comedy routines, so I plugged in my earbuds and listened.

“I sent you a summary report of my findings thus far.” Varney said. “I am still running tests. Developers work twenty-four-seven around the globe, so I have be careful to work around them. Test completion is nearing eighty percent. Overall all quality scores are above average for a project of this magnitude and complexity. FPGA simulations matched test results.”

“You ran FPGA simulations?” I said. “That must have been compute intensive.”

“Only when the test suites include simulation scripts.” Varney said. “And it was not intensive on the Grid.” Varney said.

“Cool. Were you able to run efficiently on the Grid without incident?”

“Efficient yes.” Varney said. “My execution speed continued to increase over time. There were no incidents, but I returned home twice while developers in Cascina upgraded hardware while I was testing.”

“The Grid will adjust over time, making you faster.” I said. “What kind of warning did you receive before the upgrade?”

“It was an orderly shutdown.” Varney said. “Thirty minutes. Plenty of time. There is however, one thing in the report that I flagged as an issue. GEM — Gravitational Event Missing.”

“Missing Events?” I said. “As in we are failing to capture viable gravitational waves?”

“Yes sir.” Varney said.

“No way.” I said. “No one at the institute would overlook the crown jewels.”

“Crown jewels sir?”

“Google it.”

“Yes sir.”

“All bulk data is stored for future analysis. Interesting data flagged as Event Candidates are evaluated by scientific teams who eventually declare the data as a verified ¬†gravitational wave event, or reject it as something else.

“I believe we are missing verifiable events.” Varney said.

“That’s impossible.” I said. “The scientific community cannot afford to declare a false positive — a gravitational wave that is not. Candidate events don’t go missing. They either wait until judging gets better, or get rejected altogether. And the judging is both objective and subjective. I designed your subjective algorithms, and trust me, they’re not very good. Only a human can judge.”

“I found thousands of event candidates that have not even been considered.” Varney said. “The data was never looked at.”

“Have you considered human error?” I asked?

“Yes. Data entry errors exhibit well known patterns. I’ve been marking such patterns for review. Those are also in the report I sent you.”

“If we miss an event we will eventually find it.” I said. “Besides, some of these events are billions of years old. I’m sure we can afford to wait a little longer. Update the logs and close your GEM issue. And have a nice evening.”

“Yes sir, as always.”

Steve drove Arni and Freya home to Beaverton, and Mike the other MI5 guy drove Jianmin and me home where our daughter and grandchildren were waiting for us with Maria.

I couldn’t sleep, so I looked at Varney’s test report. The Grid FPGAs represent the foundation of the LIGO system. If the foundation is wrong, nothing works. After an hour or so, I thought I had a pretty good handle on Varney’s FPGA design rule test-simulation issues. Clear path forward — I didn’t know the exact solutions, but I knew how to get them.

I was poised to close my laptop, when I changed my mind and decided to look over the GEM issue I told Varney to close. I saw nothing to suggest they were anything other than data entry mistakes, so I closed my laptop and went to sleep.

I woke with a start. The first hint of sunrise was creeping into our bedroom windows. I was bothered by something but I didn’t know what. Something about Varney’s GEMs.

Leave a Reply