RSS

Category Archives: engineering

Workforce Reduction

In world at war, something much closer to home has happened. On Monday, March 11, 2022 everything changed for me. For the first time in my life, after 25 of engineering, I was laid off. At later date, I will detail who I was working for, once the severance check goes through and I have a new position. I worked at company X for 7 years as a Design Test Engineer, an interesting title for a job that, quite honestly, got really boring at the end. From here on out, this is will a cautionary tale for anyone who wants to try Scrum.

Before I detail what the heck scrum is, I want to preface that when I started working at company X, scrum was just starting to be used in the east coast facility, and not back here in Illinois. I enjoyed what I was doing and was almost done with a complete set of code for a new test site, then Scrum started. As a spoiler, Scrum was originally meant for software programming, not hardware. Sadly, 90% of what company X made was hardware with a software component. It does not work well, if at all, or worst case, as I saw, actually slows down hardware development to an absolute crawl or even backwards. Scrum does not work. It hides the failings of bad or mediocre engineers and pulls down good or great engineers.

Scrum is based on the book SCRUM The Are of Doing Twice the Work in Half the Time, or I like to call it, the book of lies, from Jeff Sutherland. The ‘old’ way of doing things is the waterfall, when all the steps are planned out, with dates and milestones. It is not a perfect system, but in my opinion, it still works for HARDWARE based development. A modified waterfall with some agility built in to deal with setbacks has been the best compromise. I am sure some version of Scrum/Agile could work, but you really need to have it across the company. The biggest fault I have with of the book is the vilification of task switching along with the underestimating the impact non-value task that the rituals of Scrum.

From the very beginning of the Scrum switchover at company X my biggest concerns were the time to be used for the rituals and lack of true cross functional teams. Let me tackle them one by one:

Rituals and the Time used:

-Daily Stand up -15 minutes a day, ideally. In reality, depending on the size of the team and globs onto the team, this could be as long at 30 minutes. You are supposed to state what you worked on, what you will be working on, and if there is anything in the way. In reality, the sales guy chimes in how angry the customer is, the product owner, which echos the same and outside work that may be needed, and production…and quality. Instead of a nice, tight 7 person team, you end up with a 15+ person standup.

-Demo preparation -Up to an hour meeting + time it takes to produce pointless power power slides, which took me 2 hours or so per ‘sprint’.

-Demonstration – Usually an hour to 1.5 hours.

-Retrospective – Was somewhat quick at 30 minutes.

-Sprint planning – 30 minutes to an hour.

A ‘sprint’ is a 2 week period of work where we tracked information on a Jira board. Lets add up all the time spent on this: Lets be generous:

20 minutes per stand up x 10 = 200 minutes

Demo Preparation 180 minutes

Demonstration 90 minutes

Retrospective 30 minutes

Sprint Planning 30 minutes

———-

530 minutes, 8.883 hours per two week sprint. So, we more or less wasted over an entire day PER PERSON for 80 hours. That’s about 11% of our time spent doing. Meetings, planning, demos, retrospective. How is this really an improvement over the typical way of doing things? In addition, this ignores the task switching time that is supposed to be the problem with waterfall management. Even this loss of time is nothing compared to the total stall of progress I experienced. In the software world, if there is an issue, you can just throw out the code, re-write, test, and release, all in the span of 2 weeks. This is just not possible with hardware. You have to order components from external suppliers. If the design is wrong, you have to re-spin, retest, and try again. This takes weeks and weeks, months or more. You can’t just fix a few lines of code. With all the delays, the tasks just keep getting kicked into the ‘backlog’. The can just keeps getting kicked down the road. This is supposed to be impossible in scrum, but in the real world, this is not the case. The Scrum book is deceptive in a major way, summed up in a diagram.

Did you notice anything odd with this diagram? The only reason the Scrum team is faster is the boxes are smaller. In the real world, the boxes are small, but you end up with MANY more small boxes and if the task does not get complete, it had to spill over into the next sprint. The task does not get done because of the very nature of the speed of hardware development. This problem was further exacerbated in company X by repeated, intermittent failures and test cycled that could take days or even weeks to see if a change worked. Even if everything worked perfectly, the text process took close to 2 weeks to complete. You can see here what I experienced with Scrum. The original diagram also does split up the tasks into the 2 week sprints, with all the associated overhead, but does not duplicate any sprints or backtrack.

To expand my real world experience, there were 3 cable assemblies, each different, that got mated to another assembly, not interchangeable, that got mated to yet another larger system, so lets call them A, B, and C. What I ended up with A1, design the cable, 1 sprint, have external vendor make the cables, 3 to 4 sprints, 6 to 8 weeks, receive cables, test, Sprint 5. Find the cables don’t work, go back to Step A1, another sprint 7, send back to cable vendor, sprint 8,9, and 10, receive back, test, Sprint 11, initial test so we integrated in to B. Attach to B in task B1, Sprint 12, then test, Sprint 13, find there is a issue, go all the way back to A1, redesign able, Sprint 14, get new ones from vendor, sprint 15, 16, 17, skip to B1, test again, B2, Sprint 18, find it fails again…this repeated for almost a year. For three cable assemblies. It was insane, and this was just part of the whole, and you can imagine the rest was not much better. Repeat this for 5 years.

At the end of the day, the project X was no closer to being complete than it was 5 YEARS before and I am out of a job, along with Mr. M, who founded company X, was outed, sorry, retired at the same time.

From my perspective I think this is what happened: Mr. M was told about Scrum and how it was the best thing since sliced bread. He directed a SOFTWARE team to work using Scrum and it worked well, since it was software. Around five and a half years ago, Scrum was rolled out over company X. I remember having to read the book, then even been flown out to New England to learn more about Scrum. Looking back at the financial records of company X, the very same fiscal year they went from a somewhat profitable company to losing money. Over the last 6 years, company X has lost an average of $10,000,000 a year and close to $60 million total. The whole scrum process is a fiasco. This problem was not limited to the team I was on, but also with a research and development scrum team that has produced no working product despite millions of dollars of equipment purchased and external contracts. In the end, an external investment firm must of taken a close look at the finances and realized company X could be turned around and elected a new board of directors, which then removed the Mr. X and downsized the company around 10%. I was in the 10%. I know why they did it, I am still angry and disappointed, but it had to be done. If they really want to turn around company X, they need to end scrum and end it now.

Why did I write this long ass blog post? I wanted to get everything down before it fades in my memory. My prediction for company X is either they will sell the intellectual property rights to a competitor or sell the division to another company or possibly spin it of as its own company.

 

Posted by on March 19, 2022 in engineering

Leave a comment

Looking into the hyperloop.

This is a work in progress, so check back for updates!

I wanted to address the white paper produced by Elon Musk in 2013. It is an interesting concept, that several companies are actually perusing, but I am concerned with technical, economic, practical, and psychological implications of the system. As a concept, it has a strong appeal, aircraft speed on the land. From a technical perspective, in this engineer’s opinion, it poses technical challenges rivaling or exceeding landing on the moon. The economies outlined in the white paper are optimistic, at best, and I will attempt to compare the prices to real world transportation systems. The practical application of this technology appears to ignore an ever present issue in mass transit, the last mile problem. Finally, there are myriad psychological issues, including claustrophobia, uncomfortable acceleration and deceleration, and, most importantly, the American’s concept of freedom in transportation.

First, I would like to examine the technical issues. I will quote sections of the white paper in their entirety.

Earthquakes and Expansion Joints

A ground based high speed rail system is susceptible to Earthquakes and needs frequent expansion joints to deal with thermal expansion/contraction and subtle, large scale land movement.

By building a system on pylons, where the tube is not rigidly fixed at any point, you can dramatically mitigate Earthquake risk and avoid the need for expansion joints. Tucked away inside each pylon, you could place two adjustable lateral (XY) dampers and one vertical (Z) damper.

These would absorb the small length changes between pylons due to thermal changes, as well as long form subtle height changes. As land slowly settles to a new position over time, the damper neutral position can be adjusted accordingly. A telescoping tube, similar to the boxy ones used to access airplanes at airports would be needed at the end stations to address the cumulative length change of the tube.

These three short paragraphs are the first example of hand-waving that belittle the technical challenges ahead.  In the second paragraph, it is evident that Mr. Musk is not familiar with ‘Sun Kinks’ which are an issue that conventional railroads have to deal with.  The ‘Tucked away inside each pylon’ dampeners is what is know as track ballast in the railroading world.  All the rock they lay beneath railroads is not there for decorative reasons.  The tubes that would be used will not be pre-stressed and regardless, will undergo thermal expansion and contraction.

The third paragraph mentions accordion style sections of tube to take up changes in overall length.  Two major issues here:

1) The technology to create an expanding accordion structure that can withstand the pressure differential between the proposes internal near vacuum of the tube, at 100Pa and normal air pressure of 101.325kPa, or 14.7PSI, so the outside pressure is 1013 TIMES greater than the interior pressure, does not exist.  I don’t even know if it is even possible.

2) The sheer magnitude of thermal expansion is not represented here.  The distance between Los Angeles and San Francisco is 381.9 miles, following the I-5 corridor, which is the proposed route.  During a typical day, even in LA, a 25°F temperature change is a normal day.  So:

The calculation for thermal expansion is pretty easy: amount of thermal expansion = length of material * change in temperature * thermal expansion coefficient.  In this case, we will use steel, which is 0.0000065 (inch/inch/degree F).  Which seems like a small number, until you plug the numbers in:

Thermal expansion=381.9miles*5280feet*25°F*0.0000065*12=327.67 feet.  For those keeping track at home, this over 100 yards, a football field, with the end zones.  The accordion at each end would have be over 150 feet long to deal with a typical day’s temperature swing! Over the course of a year, you are looking at 2 to 3 TIMES that change.  As an example:

Another proposed route is between Kansas City and St Louis, MO.  250 miles.  So 250miles*5280feet*75°F(seasonal swing)*0.0000065=643.5 feet over the course of a year!  Just under an eighth of a mile, or a city block.

Even today, 200 years after the invention of railroads, engineers are still battling thermal expansion of steel rails.  The same steel that will be used in the proposed Hyperloop.  The next time your ride the rails, the click-clack you hear is one small thermal expansion joint after the next.

This problem alone is not insurmountable, but will require the design of a new type of vacuum proof expansion joint, which does not exist.  The proposed solution of a freestanding tube system will not work.  Tunneling would be more feasible, but very expensive.

The document now turns to the more technical section:

Reading into the white paper, I realize it will be impossible to separate the technical, economic, practical, and psychological issues with the system, since the original white paper mixes them as well, so from this point forward, I will address the issues in the order they were laid out.

This first major error that I foresee is capacity:

The Hyperloop is sized to allow expansion as the network becomes increasingly popular. The capacity would be 840 passengers per hour which more than sufficient to transport all of the 6 million passengers traveling between Los Angeles and San Francisco areas per year. In addition, this accounts for 70% of those travelers to use the Hyperloop during rush hour. The lower cost of traveling on Hyperloop is likely to result in increased demand, in which case the time between capsule departures could be significantly shortened.

These numbers didn’t seem all that realistic to me, so I went to see how many people actually travel between LA and San Francisco.  By air, it is 10,500,00, by car, 290,000 VEHICLES PER DAY, or 105,850,000 trips per year on I-5, with the caveat that the 290,000 per day is vehicles, with 1 to 6 people per vehicle.  Assuming 1.5 people per vehicle, you are looking at something like 150,000,000 people per year, for just cars + 10,500,000 by plane.  160,500,000 passengers per year/365/24=18,321 people PER HOUR, assuming 24/7, evenly spread out.  The Hyperloop could handle 4.5% of the average traffic.

The second issue is the acknowledgement of rush hour needs.  840 passengers an hour would be woefully inadequate. Just in Chicago, for just the O’hare stop for the Blue Line, there were just shy of 4,000,000 passengers.  For the O’hare stop, during rush hour, they average 2684 people an hour during the MORNING rush hour.

 

 

 

Posted by on October 30, 2018 in engineering

Leave a comment