[iaoa-advisor] FOIS and CORE

Wed Apr 21 07:19:02 CEST 2021

Dear Laure,

Some additional information below
(just received)

best,
Giancarlo

Providing additional info for CORE rankings
<http://groups.google.com/a/core.edu.au/group/rankings/t/b93f7853f21ae669?utm_source=digest&utm_medium=email>
Lin Padgham <lin.padgham at rmit.edu.au>: Apr 16 04:19PM +1000

Dear All,
We have now organised a web interface to submit additional information for
conferences whose ranks are proposed for change, in particular for
conferences where decisions were made based on the limited comparator
data. *The
url is at portal.core/edu.au/conf-changes <http://edu.au/conf-changes>. All
extra information must be submitted via this process.*
These pages have been adapted from parts of the original full submission
forms, but with everything made optional. Formatting and wording may be
slightly odd as a result. This provides guidance on the data that was
requested from full submissions. Any additional data/information can also
be provided in the final section, either by typing in directly, or
uploading a file.

Note that as everything (except conference name and submitter) is now
optional, sections B and C will show as completed (green) without any data
being added. You are free to answer whichever questions you wish. The
Google Scholar section can be safely ignored other than provided the GS h5
if it was not in the original submission. The most useful parts to provide
are B2.1, the start of B3, B5 and C.

Once finished you must hit submit. You will not receive any confirmation
email, but the front page button turning green indicates successful
submission. If after submitting you open for any reason, you will need to
resubmit, even if no changes have been made. Opening automatically
unsubmits.

In order to help in understanding the somewhat cryptic data report you will
have already received, it is helpful to look at the input form. I provide
also here some of the explanation that was also given to committees
regarding the PC profile figure in reports and the WPP report and excerpts.

The section (Senior) Program Committee in the reports is a profile of the
PC (or SPC if a large conference with both PC and SPC). This was obtained
by having submitters paste in the PC list from the web, and provide some
structured information about the format. This was then processed by our
system to obtain the Elsevier h-index of individuals, providing the bar
chart profile in the left hand figure. We then extracted the more senior
researchers (above a certain threshold) and requested that this list be fed
into our WPP tool to see where these more established researchers on the PC
were publishing. The right hand figure shows 2 whisker plots: the LHS is
the distribution of the h-index of all PC members, showing amongst other
things the median h-index. The RHS whisker plot is a distribution of those
who were above the threshold - the more established researchers.

When the list is provided to WPP that tool (based on dblp data) shows where
the individuals in the list are publishing. The data looked at is generally
where else the established researchers on the PC are publishing, and to
what extent they are publishing in the conference they are on the PC of.
The text of the full WPP report is available at the link, while the
relevant graphs and a text excerpt are pasted directly into the report.
Note that the WPP tool has been modified and now has options to provide
journals, conferences, or combined information.

Committees also had access (in some cases) to data which we have permission
from Elsevier to use by the committees in ranking, but not necessarily to
publicize in its entirety. This data gives an analysis of citations and of
h-index of most senior author on papers, within the context of the specific
FoR code. Methodology here was to obtain for every conference in the CORE
DB (where possible) the data for each paper in 2017. Conferences were then
partitioned into their FoR code areas and citation centiles were calculated
within each area for the top 1%, 5%,10%, 25% and 50%. These were also
calculated for the subset of A/A* conferences within a FoR code. By looking
at the % of papers from a given conference, compared to these centiles (in
particular the A/A* ones), we could then establish how well an individual
conference fit the average A/A* profile for that FoR code.
We used a similar approach to look at the relative strength of those
publishing in a given conference, under the assumption that if many papers
are by strong researchers, it is likely to be a strong venue. For this
calculation each paper was assigned the h-index of its strongest author,
and then centiles were calculated as above. In reports the data most often
referenced is the % of papers in a venue at the 25th A/A* centile. If this
is much lower than 25%, it is attracting fewer authors with a strong
h-index than the average A/A* conference in that FoR code. This analysis
was not available for some conferences at the point that reviews were
processed.
As with all data, this is one piece that must be considered along with
other relevant aspects. Committees attempt to form a holistic view based on
all available information, both objective data and subjective
experience/knowledge.

We welcome additional data in situations where community members who have
not made a submission on a given conference, consider that they have
additional useful information to add, for consideration by the committee.
This is particularly important in cases where reviews have been conducted
based on more limited comparator data provided by submitters, on
conferences they consider less strong than the submission conference.

It is important to remember that, to its best knowledge, CORE only includes
conferences that are legitimate academic venues with peer review of full
papers prior to acceptance. Consequently all included venues (including C
and unranked) are considered sound academic events. B conferences are "good
to very good" venues, well regarded in their area, while A is a much
smaller number of venues regarded as excellent. A* is reserved for flagship
conferences in a given area. In general only the most well known top
conference of an area will be considered a flagship. If there are multiple
top conferences in the same area, only the best will be considered a
flagship, unless they are all at about the same level. Not all sub-areas
within a FOR code, will have A* conferences, and it may well be that an A
conference has equally high quality papers as an A*, but has somewhat less
prestige, impact or reach than an A* covering a similar area. Naturally
borders are not well defined and there will always be some disagreement
around the edges. Committees attempt to be as fair and objective as
possible, whilst also avoiding everything percolating to the top.

*We welcome additional information on proposed changes, due by midnight
25th April AoE.*
*Please make submissions only via the provided interface at
http://portal.core.edu.au/conf-changes
<http://portal.core.edu.au/conf-changes>.*

Regards,
Lin (CORE rankings coordinator)

Lin Padgham
Professor in Artificial Intelligence
Computer Science, RMIT University
Melbourne, Australia.

On Sun, Apr 18, 2021 at 11:55 AM Giancarlo Guizzardi <gguizzardi at gmail.com>
wrote:

> Dear Laure,
>
> Sorry for the delay.
> Please find it in the attachment
>
> best,
> G
>
> On Thu, Apr 15, 2021 at 7:45 PM Laure Vieu <vieu at irit.fr> wrote:
>
>> Dear Giancarlo,
>>
>>
>> Thanks for this.
>>
>> If you've got the attachment, could you please forward it?
>> (I do not have access to the Google group linked)
>>
>>
>> Best,
>>
>> Laure
>>
>>
>> Le 15/04/2021 à 09:04, Guizzardi Giancarlo a écrit :
>>
>> Dear all,
>>
>> Some additional information on CORE
>>
>> best,
>> Giancarlo
>>
>> Poviding additional info for CORE rankings
>> <http://groups.google.com/a/core.edu.au/group/rankings/t/b93f7853f21ae669?utm_source=digest&utm_medium=email>
>> Lin Padgham <lin.padgham at rmit.edu.au>: Apr 14 07:52PM +1000
>>
>> Dear All,
>> For those wishing to supply additional information for review of the
>> currently recommended changes to conferences in the CORE ranking, I attach
>> pdfs of 2 sections of the full submission that was requested for new
>> conferences, those wishing to be considered for upgrade, or those listed
>> for review. These indicate some information which will be useful to the
>> committee, particularly for those conferences that were evaluated on the
>> basis of a comparator report only.
>>
>> I am currently investigating whether we can provide a web interface for
>> submission of information (and if we are able to do that, this will be
>> compulsory as it greatly assists in getting all the information into the
>> DB
>> and out to committees). I will post by the end of the week regarding how
>> information should be provided. In the meantime these attachments provide
>> some guidance as to what information will be of value.
>>
>> I do note that the section D2 where it asks for a list of at least 20*
>> top *people
>> in the area, and a report run through the WPP tool, is only of value if
>> the
>> list of people is chosen in a truly objective, repeatable and verifiable
>> fashion, and is without implicit or explicit reference to the conference
>> being evaluated. If these instructions are not followed, the report is of
>> little value in this process.
>>
>> Similarly, in section C5, the list of top people regularly involved in the
>> conference is only of value if some information is provided as to why you
>> consider them a top person in the area, as well as their google scholar
>> h-index. Just providing their university affiliation is of little value.
>>
>> Any additional supporting information can also be provided in any format
>> desired. However it is important that information is independently
>> verifiable data of some sort, and not simply opinions. While no single
>> type
>> of data is appropriate as a means of ranking, and a holistic view must be
>> sought, there should always be some objective support for views that
>> contradict what has been decided on the basis of data currently reviewed.
>>
>> If you are not planning to potentially provide further information but are
>> just wanting to know why particular decisions were made, can i please ask
>> that you wait until this information is uploaded to the DB along with the
>> new rankings. At this stage there will be a more complete decision report,
>> accounting for any extra information received. Also, it is overwhelming
>> for
>> me to try and respond to everyone, so I would greatly appreciate patience
>> until the final reports are uploaded, except for cases where people feel
>> that they may be able to contribute additional useful information.
>>
>> Regards,
>> lin (CORE rankings coordinator)
>>
>>
>>
>> Lin Padgham
>> Professor in Artificial Intelligence
>> Computer Science, RMIT University
>> Melbourne, Australia.
>> ph. +61 3 9925 3214 (but email works better)
>> ------------------------------
>> *From:* Maria Keet <mkeet at cs.uct.ac.za> <mkeet at cs.uct.ac.za>
>> *Sent:* Wednesday, April 14, 2021 4:00 PM
>> *To:* Anthony Cohn <A.G.Cohn at leeds.ac.uk> <A.G.Cohn at leeds.ac.uk>;
>> Guizzardi Giancarlo <Giancarlo.Guizzardi at unibz.it>
>> <Giancarlo.Guizzardi at unibz.it>; Barry Smith <ifomis at gmail.com>
>> <ifomis at gmail.com>; Laure Vieu <vieu at irit.fr> <vieu at irit.fr>
>> *Cc:* Nicola Guarino <guarino at loa.istc.cnr.it> <guarino at loa.istc.cnr.it>;
>> João Paulo Almeida <jpalmeida at ieee.org> <jpalmeida at ieee.org>; John
>> Bateman <bateman at uni-bremen.de> <bateman at uni-bremen.de>; Stefano Borgo
>> <stefano.borgo at cnr.it> <stefano.borgo at cnr.it>; Antony Galton
>> <A.P.Galton at exeter.ac.uk> <A.P.Galton at exeter.ac.uk>; Janna Hastings
>> <janna.hastings at gmail.com> <janna.hastings at gmail.com>; Heinrich Herre
>> <heinrich.herre at imise.uni-leipzig.de>
>> <heinrich.herre at imise.uni-leipzig.de>; Werner Kuhn
>> <werner.kuhn at gmail.com> <werner.kuhn at gmail.com>; Riichiro Mizoguchi
>> <mizo at jaist.ac.jp> <mizo at jaist.ac.jp>; Mark Musen <musen at stanford.edu>
>> <musen at stanford.edu>; Leo Obrst <lobrst at gmail.com> <lobrst at gmail.com>;
>> Barry Smith <phismith at buffalo.edu> <phismith at buffalo.edu>; Zena Wood
>> <Z.M.Wood2 at exeter.ac.uk> <Z.M.Wood2 at exeter.ac.uk>; Roberta Ferrario
>> <roberta.ferrario at loa.istc.cnr.it> <roberta.ferrario at loa.istc.cnr.it>;
>> Michael Gruninger <gruninger at mie.utoronto.ca> <gruninger at mie.utoronto.ca>;
>> thomas.studer at inf.unibe.ch <thomas.studer at inf.unibe.ch>
>> <thomas.studer at inf.unibe.ch>; IAOA Executive Council
>> <iaoa-council at ovgu.de> <iaoa-council at ovgu.de>
>> *Subject:* Re: FOIS and CORE
>>
>> Dear Tony, Giancarlo, All,
>>
>> On 14/04/2021 15:45, Anthony Cohn wrote:
>>
>> An interesting discussion and of course I agree  it’s worth making a case
>> against the “demotion”.  I agree about the h-index point – scopus has a
>> notion of “field weighted impact factor” – which adjusts for the size of
>> the community. I don’t know if scopus computes this for conferences, but
>> given the relatively small size of our community this might be an argument
>> to make?
>>
>> agreed, hence the "a need to spin that story better" note in my email.
>> afaik, it is something that they consider, but I'm not privy to all those
>> details.
>> and indeed, it's people beyond the authors' standings and their repeat
>> participation, including those of the event organisers and participants,
>> among others.
>> but as long as we don't know the reasons by it got downgraded, it's
>> guesswork why exactly
>>
>> Regards,
>> Maria
>>
>>
>>
>> Best wishes      Tony
>>
>>
>>
>> *From:* Guizzardi Giancarlo <Giancarlo.Guizzardi at unibz.it>
>> <Giancarlo.Guizzardi at unibz.it>
>> *Sent:* 14 April 2021 14:26
>> *To:* Maria Keet <mkeet at cs.uct.ac.za> <mkeet at cs.uct.ac.za>; Barry Smith
>> <ifomis at gmail.com> <ifomis at gmail.com>; Laure Vieu <vieu at irit.fr>
>> <vieu at irit.fr>
>> *Cc:* Nicola Guarino <guarino at loa.istc.cnr.it> <guarino at loa.istc.cnr.it>;
>> João Paulo Almeida <jpalmeida at ieee.org> <jpalmeida at ieee.org>; John
>> Bateman <bateman at uni-bremen.de> <bateman at uni-bremen.de>; Stefano Borgo
>> <stefano.borgo at cnr.it> <stefano.borgo at cnr.it>; Anthony Cohn
>> <A.G.Cohn at leeds.ac.uk> <A.G.Cohn at leeds.ac.uk>; Antony Galton
>> <A.P.Galton at exeter.ac.uk> <A.P.Galton at exeter.ac.uk>; Janna Hastings
>> <janna.hastings at gmail.com> <janna.hastings at gmail.com>; Heinrich Herre
>> <heinrich.herre at imise.uni-leipzig.de>
>> <heinrich.herre at imise.uni-leipzig.de>; Werner Kuhn
>> <werner.kuhn at gmail.com> <werner.kuhn at gmail.com>; Riichiro Mizoguchi
>> <mizo at jaist.ac.jp> <mizo at jaist.ac.jp>; Mark Musen <musen at stanford.edu>
>> <musen at stanford.edu>; Leo Obrst <lobrst at gmail.com> <lobrst at gmail.com>;
>> Barry Smith <phismith at buffalo.edu> <phismith at buffalo.edu>; Zena Wood
>> <Z.M.Wood2 at exeter.ac.uk> <Z.M.Wood2 at exeter.ac.uk>; Roberta Ferrario
>> <roberta.ferrario at loa.istc.cnr.it> <roberta.ferrario at loa.istc.cnr.it>;
>> Michael Gruninger <gruninger at mie.utoronto.ca> <gruninger at mie.utoronto.ca>;
>> thomas.studer at inf.unibe.ch; IAOA Executive Council <iaoa-council at ovgu.de>
>> <iaoa-council at ovgu.de>
>> *Subject:* Re: FOIS and CORE
>>
>>
>>
>> Dear Maria and all,
>>
>>
>>
>> CORE does periodic revision in its rankings from time to time.
>>
>>
>>
>> They have already requested input from the different communities many
>> months ago.
>>
>> As a community, we missed that. I wasn't concerned about that because the
>> conference was classified as A
>>
>> (well, in theory, we could have tried to make the case for A* but that
>> would not have worked out as
>>
>> our demotion shows...).
>>
>>
>>
>> The criteria used by CORE is a mixture of objective and subjective points
>> (again, the communities
>>
>> try to make the case considering both types of indicators). FOIS does not
>> do well h-index-wise but
>>
>> that is mainly due to the fact that it is a biannual conference (I won't
>> even enter the discussion
>>
>> of how much of a bad idea is to use h-index to judge conferences...). But
>> that is the same for KR
>>
>> and KR managed to revise their initially bad evaluation. Again, this
>> requires an active community effort.
>>
>> Other conferences that are small but prestigious in their relevant
>> communities have
>>
>> managed to even talk their way up to A* (e.g., PODS). ER is trying the
>> same move.
>>
>>
>>
>> As for subjective aspects, some of the points that are taken very
>> seriously is the impact
>>
>> and scientific stature of people playing key roles in the conference (PC
>> chairs and members,
>>
>> general chairs, keynote speakers, frequent authors, etc.).
>>
>>
>>
>> I think we should react now by contacting them and defending the
>> importance of the event
>>
>> as the most important event for this community and highlighting some of
>> these points
>>
>>
>>
>> best,
>>
>> Giancarlo
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> *From:* Maria Keet <mkeet at cs.uct.ac.za>
>> *Sent:* Wednesday, April 14, 2021 10:33 AM
>> *To:* Guizzardi Giancarlo <Giancarlo.Guizzardi at unibz.it>; Barry Smith <
>> ifomis at gmail.com>; Laure Vieu <vieu at irit.fr>
>> *Cc:* Nicola Guarino <guarino at loa.istc.cnr.it>; João Paulo Almeida <
>> jpalmeida at ieee.org>; John Bateman <bateman at uni-bremen.de>; Stefano Borgo
>> <stefano.borgo at cnr.it>; Cohn <A.G.Cohn at leeds.ac.uk>; Antony Galton <
>> A.P.Galton at exeter.ac.uk>; Janna Hastings <janna.hastings at gmail.com>;
>> Heinrich Herre <heinrich.herre at imise.uni-leipzig.de>; Werner Kuhn <
>> werner.kuhn at gmail.com>; Riichiro Mizoguchi <mizo at jaist.ac.jp>; Mark
>> Musen <musen at stanford.edu>; Leo Obrst <lobrst at gmail.com>; Barry Smith <
>> phismith at buffalo.edu>; Zena Wood <Z.M.Wood2 at exeter.ac.uk>; Roberta
>> Ferrario <roberta.ferrario at loa.istc.cnr.it>; Michael Gruninger <
>> gruninger at mie.utoronto.ca>; thomas.studer at inf.unibe.ch <
>> thomas.studer at inf.unibe.ch>; IAOA Executive Council <iaoa-council at ovgu.de
>> >
>> *Subject:* Re: FOIS and CORE
>>
>>
>>
>> Dear Giancarlo, All,
>>
>> It may be of use to first find out from them why the re-evaluated it
>> differently.
>> Good quality papers isn't the only criterion they use. It's also, e.g.,
>> high performers among the authors, but I assume we pass that as well. And
>> then there's the h-index of the conference, which doesn't do well, as if
>> FOIS doesn't really have any impact, actually. Well, anyway, that's what it
>> looked like when EKAW was putting the material together, when Enrico Motta
>> was showing off his tool for computing citation metrics during the EKAW
>> steering committee meeting. Anyway, if I had to put in a bet for reason
>> why, it would be that and a need to spin that story better. Still, to be
>> sure, perhaps the EC can try to find out why from Lin first?
>>
>> Regards,
>> Maria
>>
>>
>> ----
>>
>> Dr. Maria Keet
>>
>> Associate Professor
>>
>> Department of Computer Science
>>
>> University of Cape Town
>>
>> Cape Town, South Africa
>>
>> tel: +27 21 650 2667
>>
>> fax: +27 21 650 3551
>>
>> email: mkeet at cs.uct.ac.za
>>
>> work: http://www.cs.uct.ac.za
>>
>> home: http://www.meteck.org
>>
>>
>>
>> On 13/04/2021 21:44, Guizzardi Giancarlo wrote:
>>
>> Dear all,
>>
>>
>>
>> Let me raise another concern now.
>>
>> Please see the message below from the CORE committee,
>>
>> in which they are revising the classification of many CS conferences.
>>
>>
>> As many of you know, CORE is important for computer science faculties.
>>
>> Not only directly but also because they build up into several national
>> evaluation systems.
>>
>> If the classification of a conference goes down, people will prefer to
>> send their
>>
>> best papers somewhere else.
>>
>>
>>
>> According to this revised classification, FOIS went from an A conference
>> to a B conference.
>>
>> I think there is room there for protesting this classification before
>> their final decisions
>>
>> but we need to move fast and institutionally, i.e., IAOA should do it.
>>
>>
>>
>> Best regards,
>>
>> Giancarlo
>>
>>
>>
>> CORE conference rankings (preliminary) changes
>> <http://groups.google.com/a/core.edu.au/group/rankings/t/9f437bd0cd9470bf?utm_source=digest&utm_medium=email>
>>
>> Lin Padgham <lin.padgham at rmit.edu.au>: Apr 11 11:06AM +1000
>>
>> Dear CS colleagues,
>> The CORE committees have now finished reviewing the approximately 400 new
>> and existing conferences that were part of this review round.
>> More than 50% of the conferences reviewed retained their existing rank.
>> Consistent with our previous practice, we publish here
>> <
>> https://docs.google.com/spreadsheets/d/17cnG1Vgyjdu3pGdIHrvyw6HCithPAvTeDd_YB_FVOK0/edit?usp=sharing
>> >
>> a list of all planned ranking changes, allowing a short period for any
>> additional information if community members consider there has been an
>> error of judgement.* If you wish to potentially question a ranking change,
>> please notify your intent with an email to lin.padgham at gmail.com
>> <lin.padgham at gmail.com>, with subject "Rankings <confname>" by April
>> 18th.*
>> You will then be provided with the submission and the detailed decision
>> report, referencing the data on which the decision was based. *If you
>> consider you have additional data which may change the committee's
>> recommendation, this must be provided by April 25th*. Committees will then
>> review this, prior to finalisation and upload of the new CORE conference
>> ranking.
>> Regards,
>> Lin (CORE Rankings Co-ordinator)
>>
>> Lin Padgham
>> Professor in Artificial Intelligence
>> Computer Science, RMIT University
>> Melbourne, Australia.
>> ph. +61 3 9925 3214 (but email works better)
>> (please note I only work Wednesdays)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _________________________________________________________________________
>> Msg Archives: https://listserv.ovgu.de/pipermail/iaoa-advisor/
>> IAOA wiki:    http://ontolog-02.cim3.net/wiki/IAOA
>> IAOA website: http://iaoa.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.ovgu.de/pipermail/iaoa-advisor/attachments/20210421/ff6c56cd/attachment-0001.htm>