Month: March 2020

Everywhere you go on the internet is filled with Data. If you’re looking at an image on Instagram, that is data behind the scenes displaying that image for you. If you’re reading an article on a new platform, there is data behind the scenes displaying that data as well. If you’re reading this article, this might seem very elementary for you.

This Data is used and shared throughout the web. I’m sure you know that when we want to access some data for an app or project that we are working on we will look for an API to use. However, what if we can’t find an API that is providing the specific type of information that we need. Ahhh that is where Data Scraping comes in. Now API’s basically are “scraping” data, or pulling data, from the source that they intend. But it is just another method of Data scraping. API’s working differently in the fact that an API, application programming interface, is an intermediary that allows one software to talk to another software. Basically, an API allows the user to open up data and functionality to other users and developers.

API’s are the most popular, and easiest, ways of gathering the data that you need. However, it has some downsides. First of all, an API does not have access to ALL of the data, it is very specific. Also, there are limits to the amount of times you can “call” and access this data. These rates vary from one API to another. Most of the time to get more usage out of an API you have to pay a monthly or yearly rate. Also there is an issue of legality. Now, arguably, it is not illegal to use an API and get data and is not copy-rightable. However, the database where that data is stored that you are accessing can, arguably, be copyrighted. So this is something that should be considered when using API’s.

Now the other option for getting data that you need, is Web or Data scraping. These are the best options for you for many reasons. For one, you always get up to date data. You are not relying on the providers of an API to keep their information up to date.

Second, you can customize and specify the exact data you are looking for. Sometimes an API just won’t cut it and you can’t find the right API to get the information you are looking for. So with Web Scraping you can specify exactly what you need.

Third, there are no RATE LIMITS!! You control everything. You don’t have to pay someone to make more “calls” for the data or to just gain access in general. You are implementing the scraper and getting the data yourself. (Assuming you make the scraper and don’t outsource to a web scraping service. Yes you can do this… Lazy!)

Also, you can stay anonymous. You can gather the data that you need and stay private without providing your information to a websites Administrator. With an API you must register and account and get your own key, and they track every request for data that you make. So it is practically impossible to stay anonymous while utilizing an API.

Both options, API’s and Web Scraping, have their purpose and are very powerful tools. You must decide which one will better suit the needs that you have.

Next post I will be going into detail on how to make your own web scraper utilizing a Ruby on Rails app.

I can’t believe it has been 5 years since my first blog post. And, wow… how things have changed. I originally started a blog to practice writing. I still have a desire to write a book. I keep on having more and more fictional inspirations that I will write down and store for later. Writing is difficult, and I admire anyone that can sit down and actually finish writing a book.

Besides my desire for writing a book, 5 years ago I was in a completely different… world. 5 years ago I was a new father, a CAD Designer looking to go back to school for engineering, and in a toxic relationship that I stupidly fought for, for way to long. Now I am 6 weeks in at Flatiron school bootcamp, well on my way to becoming a Software Engineer! I am married and have two kids. Life is good. How did I get here?

It’s crazy to think about the past, decisions you have made. Maybe ones that you wish you hadn’t made or ones that you’re glad you made. Either way, the passing of time can be a strange experience at times. My life has been full of its “ups and downs” so to speak. I used to be a very pessimistic person, hiding under the guise of optimism. I am always hard on myself when meeting with the helping hand of failure. I say helping hand.. This is a lesson that took me a long time to learn. Failure is one of the best experiences that life can give you. It’s how you adapt and learn from it. It is life’s Teacher.

I have had many jobs as a drafter/CAD designer, and I enjoyed it for the most part. The love for coding and problem solving that I found steered me away from my engineering ambitions. It is still a subject that fascinates me, but I don’t think it is something that I would want to do as a career.

I have also tried my hand at other things. My dad was an Air Traffic Controller. This was something that amazed me as a kid. I remember my Mom and siblings going to the Tower to meet him for lunch and being able to look at all the instruments that they use and the planes flying in. It was an incredible experience. Mind you, this was before 9/11 and then security got extremely tight, and we had to take a step back from the adventures in the “tower”. My dad encouraged me to take a test to see if I qualified for air traffic control “school”. I don’t like studying… It is not my forte. So like most tests I would take, lol, I winged it. I made a 94, which then my dad proceeded to reply, “well that isn’t a 97″…. Which was what he made on it back in the day. I think I get my sarcastic humor from my dad lol.

Anyways, long story short, even with a high test score I didn’t make the cut. This was disappointing to say the least. I was looking forward to following in my dad footsteps. From there I just continued my career within the Engineering field. I learned a lot from some amazing engineers. You might think that you are intelligent, until you work next to some truly “genius” level individuals. It was a really humbling experience, and motivating as well.

After going through a really bad break up with before mentioned toxic relationship, I was in a dark place. I had been laid off multiple times over a 3 year period, had to move back in with my parents (something that feels humiliating at my age), and lost the person, who at the time I thought was the “one”. If you are not a stranger to the feelings of depression, you can probably imagine how I felt. It took a long while, and my close friends being there for me to pull myself out of it. I remember some days sleeping for 16 hours straight and then still not wanting to get out of bed. It was a tough time.

Life throws you curve balls. You just need to learn how to hit it. There are good times and bad, but thats what makes life, well.. LIFE. After going through all of these “dark” times, I met the real love of my life. I say met, but actually I’ve known her all of my life! It’s crazy how life works out sometimes. I married my best friends older sister, who I had a crush on since I was like 7 years old. (high five!) There is a lot of history between our families. My best friend was actually named after my dad. My dad and my wifes dad were best friends growing up. I couldn’t be happier and I can’t wait to see what the future holds for us.

I am now in the Software Engineering program at Flatiron school. It is tough! It is tough, but rewarding. I have learned so much in a short period of time and I am loving every moment of this journey. I have made good friends and met some amazing people. I look forward to writing some more and keeping you apprised of my progress, accomplishments, and even failures.

~ Peace, Mitchell