ASC23: Application Results

By Dan Olds

The ASC23 organizers put together a slate of fiendishly difficult applications for the students this year. The apps were a mix of traditional HPC packages, like WRF-Hydro and FVCOM, plus machine learning centric programs like the YLLM large language model and DeepMD.

This is probably the toughest set of apps I’ve seen in a student cluster competition, and they definitely challenged the students, particularly when you factor in the limited time they had to turn in results and the need to meticulously manage the power draw of their system.

Here’s a rundown on how the top teams finished on each of the major applications:

DeepMD: AI augmented molecular dynamics modeling. Average score 7.19, median score 6.71, 18 points maximum score. This was a difficult challenge for the students, judging by the median score of 7.57. However, all the teams managed to turn in a valid result for DeepMD, which wasn’t the case for all of the competition apps.

First Place: Zhejiang University. Their single-node, eight GPU system performed very well on DeepMD, netting a total of 15.40 points, which significantly outpaced the teams with more convention hardware (and actual clusters…lol).

Second Place: Peking University, with a score of 11.87, took down second place. Did the extra GPU in their three-node, nine GPU (vs. eight GPUs for most other teams), cluster make the difference for them on DeepMD? Maybe not, the margin between second place and third place was very small at .31 points, so it was probably superior application optimization that made the difference for Team Peking.

Third Place: Lanzhou University with a score of 11.56 using their three-node, six GPU, cluster. This is one of the best results we’ve seen from Lanzhou, who is competing for the second time at the ASC event.

YLLM: A large language model task which required the students to build two models that generated 1.157 billion and 17.888 billion tokens. The average score was 7.41 points with a median score of 8.08, and an 18 point maximum score. From what I can tell, all of the teams were able to generate the smaller set of tokens but several had problems with the larger 17 billion token requirement.

First Place: University of Science & Technology of China, the home team barely topped the field with their score of 17.77 points. They were driving one of the largest clusters in the competition with their four-node, eight GPU configuration.

Second Place: Shanxi University with a score of 17.64. They were extremely close to winning this part of the competition, only fractions of a point behind the first-place finisher. The team was also running a four-node, eight GPU system.

Third Place: Peking University with a score of 14.16 which was off the pace set by the top two teams, but good enough to give them a solid third place finish over the rest of the field.

WRF-Hydro: In the simplest terms, this application models climate and water movement. But WRF-Hydro isn’t simple at all, judging by the difficulty that the ASC23 had trying to run and optimize it. The average score for this app was only 2.77 points with a median of 2.79, ouch! While most of the teams were able to turn in a result, the scores were uniformly low, which demonstrates just how challenging this application was for the ASC teams.

First Place: Peking University turned in a score of 7.97 out of a maximum of 18 possible points. This was enough to take first place, which gives Team Peking their only outright win of the competition.

Second Place: Shanghai Jiao Tong University with 6.48 points. This long-time competitor makes the application leaderboard for the first time in the ASC23 competition.

Third Place: University of Science & Technology of China with 4.31 points. The home team makes another appearance on the application leaderboard with their performance on WRF-Hydro.

FVCOM: Used for modeling water circulation and things like salinity and other factors. FVCOM was the ASC23 mystery application, which means that it was a surprise workload for the students that they couldn’t prepare for beforehand. It turned out to be the most difficult app in the competition with an average score of only 2.14 points and a median score of, get this, 0.00 points. Now that’s a tough application!

First Place: Qinghai University pulled down first place with a score of 7.14 points, which is more than 3x the average score, but well under the maximum score of 18 points.

Second Place: Tsinghua University with 6.18 points. This is the first appearance on the leaderboard for Tsinghua, the most successful student cluster competition team in history. The last few years have been rebuilding years for Tsinghua as they bring new students onto their team.

Third Place: Peking University with 5.59 points. Another solid score for Team Peking on the toughest application in the competition.

Next up, we’ll put a bow on our ASC23 competition coverage by revealing the top finishers and looking at how they put together their championship runs. Stay tuned….