Stock Prognosticators? (6/11/14)
A first of its kind study by researchers at the University of Iowa suggests Yahoo’s finance message boards have a small degree of ability to predict stock price movements.
The study, “Stock Chatter: Using stock sentiment to predict price direction,” also found that more than two-thirds of the message board comments had nothing to do with finance.
The researchers analyzed 70,000 posts by more than 7,000 commenters on Yahoo’s finance message boards from April to June 2011. They determined what sentiment, if any, they expressed about 11 Fortune 500 stocks, either bullish, bearish, or neutral. The researchers then looked at the movement of those stocks’ prices the next day. Depending on the model the researchers used to classify the statements, they found that the sentiment expressed on the message boards accurately reflected the price movement anywhere between 52 and 64 percent of the time.
Michael Rechenthin, who conducted the study as a doctoral student in the Tippie College of Business, says that while the lower accuracy figures can be attributed to randomness, the 64 percent figure is statistically significant and shows a small degree of predictive ability.
The study found the predictive ability lasted only one day, though, and disappeared on follow-up days.
The study also found that only a small number of users produced the largest number of comments, and those comments seem to be responsible for whatever predictive quality the message boards had. Rechenthin says only 3 percent of the commenters wrote 50 percent of the posts during the study period, and 11 percent wrote 75 percent of the posts.
Those extreme users also tended to be the most opinionated, he says, and when they were removed from the study, what predictive ability the boards had all but disappeared. He says the study did not determine whether this was because the most forceful opinions affected stock price, or if those with stronger opinions were more seasoned market observers and able to see trends.
The study also found that, as one might expect of public message boards, many of the posts were completely irrelevant to the topic. They determined that 68 percent of posts were things like advertisements, spam, flame wars, or comments by trolls.
Politics tended to be the most common off-topic topics, he says, with President Obama and former President Bush frequent topics of criticism. The study period also included former Rep. Anthony Weiner’s resignation from Congress after texting inappropriate photos, and many comments were related to him.
Rechenthin cautions that as this is the first study of its type and follow-up studies will be needed using larger data sets. The research was published recently in the journal Algorithmic Finance and was coauthored with W. Nick Street, professor of management sciences in the Tippie College of Business, and Padmini Srinivasan, professor of computer science at the University of Iowa.
Drafty Research (5/6/14)
When general managers gather in New York for the NFL draft this week, they’ll be awash in statistics, scouting reports, interview data, and video clips as they look for a way to digest it all and make the best draft selections for their respective teams.
How should the selection be made? They can use a rule of thumb, like filling the biggest hole in their roster, or picking the best player available. But how is best measured? University of Iowa professor Jeff Ohlmann says several teams are arming their staffs with information based on analytics in efforts to gain any edge that they can.
One of Ohlmann’s research focuses is how a sports team can optimize its draft selections, and this work has led to the development of a smartphone app to help fantasy football and baseball team owners pick their teams. Ohlmann says that a sports draft is a great example of what is called a “sequential decision-making problem with uncertainty,” an area of research that attracts interest of both academics and practitioners. For example, logistics companies need to design routes to supply goods without knowing exactly how much product will be required by the retailer or precisely how long it may take to get there.
In football, the uncertainty lies in not knowing how well a player may perform in the NFL, or even if that player will be available when a team is “on the clock” because some other team may pick him first. Using probability distributions to model the uncertainty, Ohlmann and other researchers have developed models that use mathematical techniques to maximize the expected value of the players which a team drafts. At its heart, the model tries to help general managers overcome the fundamental handicap of not knowing what players will be available to draft in future rounds.
In Ohlmann’s approach, a team projects opposing teams’ selections to forecast which players will be available to the team in future rounds. He says that while there will surely be errors in guessing which players other teams will select, the errors often cancel each other out to create a sufficiently accurate forecast of the draft as a whole. The result, he says, is a model that produces a draft strategy that typically dominates alternative drafting rules-of-thumb.
“Rules-of-thumb have flaws in them and you can do better if you try to predict what will happen in the future,” he says. “A draft strategy based on analytics isn’t guaranteed to dominate, but more often than not, it will be the best strategy.”
He acknowledges data analysis is not a crystal ball and will not be 100 percent accurate in identifying which players are going to be successful. “But it may help avoid a bad decision in cases when the front office personnel are 'fooled by their eyes’ and let emotions affect their decisions.”
Ohlmann, who teaches in the department of Management Sciences in the Tippie College of Business, has used his sports research to develop and teach a first-year seminar, Sports Analytics, to introduce analytical tools to students using a topic that many students are naturally interested in.
“My goal is to show students how to formulate sports-related questions and then use data and math to try to answer them rather than just qualitatively debating them,” he says.
Is Big Data Dating the Key to Long-Lasting Romance? (BBC News, 3/24/14)
If you want to know if a prospective date is relationship material, just ask them three questions, says Christian Rudder, one of the founders of U.S. Internet dating site OKCupid.
- "Do you like horror movies?"
- "Have you ever traveled around another country alone?"
- "Wouldn't it be fun to chuck it all and go live on a sailboat?"
Why? Because these are the questions first date couples agree on most often, he says.
Mr. Rudder discovered this by analysing large amounts of data on OKCupid members who ended up in relationships.
Dating agencies like OKCupid, Match.com—which acquired OKCupid in 2011 for $50m (£30m)—eHarmony and many others, amass this data by making users answer questions about themselves when they sign up.
Some agencies ask as many as 400 questions, and the answers are fed in to large data repositories. Match.com estimates that it has more than 70 terabytes (70,000 gigabytes) of data about its customers.
Applying big data analytics to these treasure troves of information is helping the agencies provide better matches for their customers. And more satisfied customers mean bigger profits.
U.S. Internet dating revenues top $2bn (£1.2bn) annually, according to research company IBISWorld. Just under one in 10 of all American adults have tried it.
The market for dating using mobile apps is particularly strong and is predicted to grow from about $1bn in 2011 to $2.3bn by 2016, according to Juniper Research.
There is, however, a problem: people lie.
To present themselves in what they believe to be a better light, the information customers provide about themselves is not always completely accurate: men are most commonly economical with the truth about age, height, and income, while with women it's age, weight, and build.
Mr. Rudder adds that many users also supply other inaccurate information about themselves unintentionally.
"My intuition is that most of what users enter is true, but people do misunderstand themselves," he says.
For example, a user may honestly believe that they listen mostly to classical music, but analysis of their iTunes listening history or their Spotify playlists might provide a far more accurate picture of their listening habits.
Inaccurate data is a problem because it can lead to unsuitable matches, so some dating agencies are exploring ways to supplement user-provided data with that gathered from other sources.
With users' permission, dating services could access vast amounts of data from sources including their browser and search histories, film-viewing habits from services such as Netflix and Lovefilm, and purchase histories from online shops like Amazon.
But the problem with this approach is that there is a limit to how much data is really useful, Mr. Rudder believes.
"We've found that the answers to some questions provide useful information, but if you just collect more data you don't get high returns on it," he says.
This hasn't stopped Hinge, a Washington, D.C.-based dating company, gathering information about its customers from their Facebook pages.
The data is likely to be accurate because other Facebook users police it, Justin McLeod, the company's founder, believes.
"You can't lie about where you were educated because one of your friends is likely to say, 'You never went to that school'," he points out.
It also infers information about people by looking at their friends, Mr. McLeod says.
"There is definitely useful information contained in the fact that you are a friend of someone."
Hinge suggests matches with people known to their Facebook friends.
"If you show a preference for people who work in finance, or you tend to like Bob's friends but not Ann's, we use that when we curate possible matches," he explains.
The pool of potential matches can be considerable, because Hinge users have an average of 700 Facebook friends, Mr McLeod adds.
But it turns out that algorithms can produce good matches without asking users for any data about themselves at all.
For example, Dr. Kang Zhao, an assistant professor at the University of Iowa and an expert in business analytics and social network analysis, has created a match-making system based on a technique known as collaborative filtering.
Dr. Zhao's system looks at users' behaviour as they browse a dating site for prospective partners, and at the responses they receive from people they contact.
"If you are a boy we identify people who like the same girls as you—which indicates similar taste—and people who get the same response from these girls as you do—which indicates similar attractiveness," he explains.
Dr. Zhao's algorithm can then suggest potential partners in the same way websites like Amazon or Netflix recommend products or movies, based on the behaviour of other customers who have bought the same products, or enjoyed the same films.
Internet dating may be big business, but no one has yet devised the perfect matching system. It may well be that the secret of true love is simply not susceptible to big data or any other type of analysis.
"Two people may have exactly the same iTunes history," OKCupid's Christian Rudder concludes, "but if one doesn't like the other's clothes or the way they look then there simply won't be any future in that relationship."
Dating Algorithm Makes Matches (KGAN, 2/14/14)
One of the hardest parts of dating is the uncertainty: will the person you like, like you back?
Finally, University of Iowa researcher Kang Zhao figured out an algorithm that matches up users with dates based on the likelihood that your crush will like you back.
"This algorithm can potentially find you a good date," Zhao said. "But whether it's a perfect match, whether you find your soul mate, you'll probably have to take it offline."
Zhao already has that figured out—he's been married to his wife, Tracy, for seven years. But, to do his research, he had to sign up for dating websites, too, something he had to clear at home, first.
"I just have to tell her, 'OK, I'm going to study this. I still love you. I still love our family,'" he said.
Zhao's algorithm is public domain, and he has already been contacted by some smaller dating websites to use it as a part of their platforms.
Machine Learning + Love (WNYC, 2/12/14)
Log onto an online dating site and you are asking a machine for romantic assistance. That's cool, but you might as well understand how it works, right?
There's an algorithm picking and choosing which profile to put in front of which users, and sometimes it works—roughly a third of marriages these days begin online—and other times it doesn't. On this week's New Tech City, host Manoush Zomorodi tracks down some smart people who are writing and improving the matching systems of dating sites.
Kenneth Cukier, data editor at The Economist, explains "you'd be a fool to try to do online dating without machine intelligence, without machine learning." So we get him to explain what that means.
Kang Zhao, assistant professor of management sciences at the University of Iowa, is a very smart guy who has a plan to make sure the matches in front of you are people you'd actually like, and who will actually respond to your messages. "There are ways to improve [profiles] because the information you have in your profile is sometimes just too much."
And then we put all this to someone responsible for a whole lot of online meetings, V.P. of matching for eHarmony, Steve Carter, who says a few unexpected things, including that dating sites only work if you shake up your rigid mindset and embrace the real life, offline magic of face-to-face dating.
UI Developed Algorithm Gives Online Dating a New Spin (KCRG, 2/5/14)
It's like Netflix, but for dating. A new algorithm at the University of Iowa is giving online dating a new spin.
The currently unnamed design matches users with compatible partners uses data about the person's past dating habits and levels of attractiveness.
The method could be compared to the popular on-demand Internet streaming site, Netflix. It suggests movies to users based on their previous film and television choices.
Kang Zhao, assistant professor at the Tippie College of Business, says this method is different from any other site out there.
"So what we did is to look at your previous interaction history," he said. "We believe your actions speak louder than what you have in your profile."
The algorithm is still being developed. Zhao says he has spoken with executives at dating websites about using this new method of online dating.
Predicting Stock Prices (1/31/14)
A new study from the University of Iowa shows evidence that stock price movements are, in fact, predictable during short windows.
The study by researchers in the Tippie College of Business suggests that price movements can be predicted with a better than 50-50 accuracy for anywhere up to one minute after the stock leaves the confines of its bid-ask spread. Probabilities continue to be significant until about five minutes after it leaves the spread. By 30 minutes, the predictability window has closed.
The researchers—Nick Street, professor of management sciences, and doctoral student Michael Rechenthin—say the work questions the generally held belief that stock prices cannot be predicted. While factors like news or financial reports can move stock prices, the thinking holds, nothing inherent in a price’s trend line can be used to predict where the price goes next.
“This study is the first step in showing that there is predictability, and that once a price escapes the confines of the bid-ask spread, it’s showing a trend,” says Rechenthin, a former Chicago Stock Exchange floor trader whose dissertation looks at building models for predicting future stock price direction. “In other words, it’s more than just a coin flip where the price goes.”
The study examined price movements of a single stock—the S&P 500 exchange traded stock fund (SPY)—during 2005. The stock holds all 500 Standard and Poor’s stocks and is considered representative of the overall U.S. market. It’s also one of the most heavily traded equities on the market, with an average of more than 90,000 transactions a day during the study period, so it provides a wealth of study data.
Their analysis found no predictability of the stock’s price within the bid-ask spread—that is, the space between the price that buyers are willing to pay for a stock (the bid) and the price sellers are willing to sell it for (the ask)—as the market tries to set the value of an asset. The key to their study is what happens once traders did set a value and the price escaped that spread. Once it did escape, the study tracked the stock’s price at 1, 3, 5, 10, and 20 seconds, and 1, 5, and 30 minutes.
The study found the stock price typically broke the spread after five to ten seconds, and the predictability of its subsequent movements depended on the pattern of its most recent trades. For instance, if the stock’s two most recent trades were an uptick followed by a downtick, there was a 52 percent probability the trend reversed itself within five seconds. Within 20 seconds, it had a 43 percent probability of reversal.
Rechenthin says these trends are driven only by previous trade prices because other factors that drive price—news or financial statements—cannot be incorporated into a price in such a short window.
While a 52 percent probability may not seem like much of a better probability than 50 percent, Street points out that in the ocean of data that is stock trading, it is a notable increase, and something that can be exploited. The next step is to develop a working model that takes advantage of these probabilities for more efficient trading.
The study, “Using Conditional Probability to Identify Trends in Intra-day High-Frequency Equity Pricing,” was published recently in the journal Physica A.
UI Team's Dating Algorithm Could Mean Better Matches (Iowa City Press-Citizen, 1/5/14)
University of Iowa researchers have uncovered a curious pattern in human behavior when it comes to online dating services: People think they know what type of person they want to date, but their behavior online says otherwise.
So, how do you find the perfect match?
Kang Zhao, assistant professor of management sciences at UI's Tippie College of Business, and UI doctoral student Xi Wang have created an algorithm for dating sites that uses a person's contact history to recommend potential partners.
"Your actions speak louder than your profile," Zhao said. "Your profile might say you like tall women, but all the women you contact are short."
The algorithm takes that disconnect into account by combining two factors: a person's taste (what type of person they approach) and a person's attractiveness/unattractiveness (how many of those people reciprocate that contact).
"You have to like someone to approach them, and there must be some sort of attraction for them to respond," Zhao said. "A person's unattractiveness is reflected by those you approached but didn't get back to you. Maybe they were out of your league.
"We can recommend potential partners in the same league with you," he said.
So far, Zhao has been approached by three different online dating services interested in using his algorithm. He declined to name those services.
During their research, Zhao and Wang used data provided by a popular commercial online dating company whose identity is being kept confidential. They looked at 475,000 initial contacts involving 47,000 users in two U.S. cities over a 196-day span. The users included 28,000 men and 19,000 women. Men made 80 percent of the initial contacts.
Zhao said that according to his data, only 25 percent of those initial contacts were reciprocated. But when the algorithm is applied, those return rates jump to 44 percent.
"Your previous actions are a better reflection of your tastes," Zhao said.
Wang said he used online dating services in the past but was always disappointed with the results.
"I wasn't interested in the people they recommended," she said. "I think with our model I would get better recommendations."
The algorithm is similar to the one used by Netflix, which recommends movies based on a costumer's viewing history. Zhao said his algorithm also could be used for other services that match people with something they are searching for online, such as jobs and colleges.
Another phenomenon Zhao discovered was that his algorithm worked best for men with "athletic" body types connecting with women with "athletic" or "fit" body types, and for women who indicate they "want many kids." The model also works best for users who upload more photos of themselves.
"The more active you are" on the dating website, the better this algorithm is in selecting your partner, Zhao said. "The men who are more athletic are probably more active on the dating site because they are more confident, and the same is true for women who are more athletic or fit."
Zhao's study was coauthored by Mo Yu of Penn State University and Bo Gao of Beijing Jiaotong University. It will be published in an upcoming issue of IEEE Intelligent Systems.
- Can a New Online Dating Algorithm Transform Your Love Life? (Refinery 29, 12/11/13)
- Need Love? There's a New Algorithm for That (Jezebel, 12/11/13)
- University of Iowa Professor Revamps Online Dating with New Algorithm (Philly.com, 12/11/13)
- A New Online Dating Algorithm Will Match You With Someone You Might Actually Have A Chance With (Business Insider, 12/11/13)
- Researchers: Dating Sites Have It All Wrong (Consumer Affairs, 12/9/13)
- Why the Future of Online Dating Relies on Ignoring You (Forbes, 12/7/13)
- UI Researchers: Netflix-Style Tracking Can Increase Online Dating Success (Des Moines Register, 12/4/13)
- Love Connection (12/4/13)
- University of Iowa Professor Advises on Disaster Relief Logistics (The Gazette, 11/16/13)
- Getting Aid to Philippines Is Challenging, Costly (KCRG, 11/13/13)
- Mathematics Turned to Helping Disaster Relief After Philippine Typhoon (UPI, 11/12/13)
- Thomas Elected to INFORMS Transportation Science and Logistics Society (10/29/13)
- Pant Named Associate Editor of the Year (10/11/13)
- Discover 11 Hot College Majors That Lead to Jobs (U.S. News & World Report, 9/10/13)
- More Shipments a Good Sign for the Economy (The Gazette, 8/24/13)
- University of Iowa Offers New Business Analytics Major for Undergrads (Data Informed, 8/8/13)
- Mining Big Data (The Press-Citizen, 8/3/13)
- New Safety Regulations Limit Hours for Truck Drivers (KWWL, 7/2/13)
- Tippie Faculty Member Releases Text on Analyzing Big Data (5/20/13)
- BTA Students Attend AITP-NCC (4/8/13)