This directory contains a couple of "helper" tools for API int-to-enum conversion.
This is partly done by scraping Android API reference, but it's mostly done by human.
The results are to be saved as "map.ext.csv" and "methodmap.ext.csv" that are to be added to "map.csv" and "methodmap.csv" in Mono.Android source.
* "doc scraper" part
With this approach, we can colledct possible enum values in generic approach.
Pros:
- It could reduce massive manual edits.
Cons:
- It is not automatically doable to map enum types to methods, and this cannot be
done without "filtering out" "already-done" methods.
- There are lots of arbitrarily named enum types that cannot be automatically mapped.
Required tools will be described below (some are done, but not limited to them).
** Const lookup
*** generate-const-list.exe
This tool collects "Constants" from the SDK reference pages into XML.
The result is printed to stdout, stored as const-list-*.xml by Makefile.
** Method argument lookup - not really in practical use
This approach has never been used, and won't be in use because it is
now much easier to review methods and properties using manually-
maintained list.
*** method lookup
- for each constants listed in const-list-* XML, grep the constant name as in URL form
and collect "which documents mentions the value" list. Key is document, value is
a list of enums. This is done for efficient document parse.
- For each key doc, load and get relevant methods, then store the relation map into
another XML, keyed by each method.
*** generate-enumlist-to-query.exe
This tool generates a flat list text from const-list-*.xml for input to other tool.
The result is printed to stdout, stored as intermediary-enum-list.txt by Makefile.
*** generate-intermediary-doc-enum-mapping.exe
This tool generates "which documents mention the value" mappings.
WARNING: this may take many hours to complete. You had better run this tool only when
a new API Level is out.
It takes reference doc and intermediary-enum-list.txt.
The result is printed to stdout, stored as intermediary-doc-enum-mapping.tsv by Makefile.
*** generate-intermediary-enum-candidates.exe
This tool looks up types that have more than one fields that share prefixes, which is,
substring before '_'. For example, there are DEFAULT_KEY_DIALER and DEFAULT_KEYS_DISABLE
in android.app.Activity, hence android.app.Activity matches this condition with "DEFAULT_".
*** generate-const-mapping.exe
This tool converts <enum-conversion-mappings><map> elements into map.csv.
The <map> element can be half-automatically generated by generate-intermediary-enum-candidates.exe.
TODO: I have added "merge" attribute that indicates "merging" the specified enum values into current enums.
This is required for implemented interface consts such as ContactsContract$CommonDataKinds$Email
for EmailDataKind that should be merged with ContactsContract$CommonDataKinds$CommonColumns BaseDataKind.
* C# code lookup with blacklisting
With this approach, we collect "public const int" fields and examine if they should
be turned into enums, and "blacklist" those constants that should not become enum.
** lookup
I have identified some namespaces and types that should not be included in the
results. Therefore, this lookup can be done as:
grep "const int " Mono.Android/platforms/android-14/src/generated/*.cs \
| grep -v Javax.Microedition \ # opengl
| grep -v Dalvik.Bytecode \ # dex opcodes
| grep -v Android.Resource.cs \ # R.* fields that has to be kept as int
| grep -v BluetoothAssignedNumbers \ # describes several companies, listed just "examples", could be extended or take custom values.
| grep -v GLES \ # android.opengl
| grep -v ContentsFileDescriptor \ # Parcelable.describeContents() return, used by many types
| grep -v ParcelableWriteReturnValue \ # Parcelable, flag without any other member
Then we lookup methods that contains int argument or int return value. This can be
looked up "public", "int", "(" and ")", except those which contain "const" (otherwise
this search results will contain consts).
grep "public" | grep -v "const" | grep "int" | grep "(" | grep ")" \
(exclusion follows)
The results were saved as remaining-int-consts.txt. We stored past
final resultsper API level (since 15, as the original work started
as of API Level 14 and versioning came up after that level).
** examine consts and methods
It is easier to begin with constaints with many values.
generate-const-list.exe output will be helpful.
** blacklisting
First to note: this part is not helpful for automated tasks.
They are rather historical records and we don't even read them
nowadays, so they would be helpful only when you are curious
"why this is not enumified...?"
Field blacklist is described in either namespace, class or individual field name.
Then build input to "uniq" as in follows: grep namespace(,class(,field)) from the
first grep results.
(This is not really *automatically* done in the actual enum conversion
work; the list is maintained manually.)
* verification
There is no automatic verification (at least yet).
Though there will be some verification aids:
** reduction_rules.txt
"make remaining-int-methods.txt" creates the text that contains
int and '[(\[]' from the sources (in API LEVEL 16).
copy it to some dummy file and open vi.
Paste reduction_rules.txt contents, which run lots of replaces.
(could be sed script...)
grep "int[\[ ]" against the output file and get filtered results.
add methodmap.ext.csv entries based on the remaining lines above.
** When new API level arrives ...
(updated: this description was initially written as of 14 as the
latest API version, then slightly updated when 15 actually arrived.)
Right now remaining-int-methods.txt is the source to apply filter.
When the new API level arrives, it should be able to create a
diff between the "remaining" lines between new and old API levels.
So, get the diff and use this for filtering input. But beware,
the filters should be re-examined as those filters are examined
through human check e.g. "OK, there is no enum-convertible "maxBlah"
in the remaining methods, I reviewed them".
When making changes in mappings, make sure that the API does not
"vanish" due to wrong mappings. Wrong reduction rule or wrong mapping
path (package/type/method/parameter) do not harm much (as it will
just fail to reduce methods to review), but if you have wrong
destination type, it will just disappear.
Corcompare significantly helps reducing this kind of mistakes much.
For constants and fields, it's easier as we have far less lines
to check, and we store the remaining lines in API level 14 in git.
This is a diff between 14 and 15, for example:
https://gist.github.com/d3df1c611055e4620d02
This is still painful, but not huge as now we have filter candidates
and we only have to review diffs, not the entire API.
*** Tasks to do, step by step
- edit Makefile and replace any *previous* version number to the latest number.
- run make
- have a look at remaining-int-consts.txt. git diff differentiates
longstanding ones and new ones. There'd be a lot new at first.
They have to be reduced to the "we reviewed all of those
constants" state.
- They can be reduced by adding more enums. Add more entries on
enum-conversion-mappings.xml, run make to get map.ext.csv and
replace additional lines in src/Mono.Android/map.csv with the
entire map.ext.csv contents.
(after "// automatically generated ones" line)
Note that "// automatically generated ones" is semantically
important and recognized by enum mapping parser to determine
whether to generate corresponding [Obsolete] field or not.
- for additional entries to good-old constants (which must NOT be
after "// automatically generated ones" line) just edit map.csv.
- then run "make API_LEVELS=[latest] clean all" in src/Mono.Android
to build enumified dll.
- note tht API Level description in the API docs are used to
become part of the CSV entry. The first item is the API Level.
That often brings problem. This is what happened when API Level
26 as stable has arrived: Google had published "android-26" API
which used to be "android-O", they still hadn't published the
corresponding docs and therefore the docs say "O" but the actual
API was "26". That later brought confusion at generator.exe ran.
To workaround this issue, replace the wrong API level specifier
(e.g. "O") with the actual API level (e.g. "26"). Use regex like
/^O/26/ to achieve this.
- go back to this directory and rebuild remaining-int-consts.txt
and check if the enumified lines disappeared.
- There are some constaints that are NOT enumified
- Some of the int consts are on blacklist-int-consts.txt.
This list also describes the reason why they are left as int.
- No need to enumify @Deprecated constants.
- There are "argurably enumifiable" constants or "enumifiable
but brings breakage" constants. They are commented on map.csv
immediately before "// automatically generated ones" line.
- Once int constants are converted to enums, the next step is to
review all the API methods to find what parameters and return
values need to be changed to those new enums. Note that it is
impossible to review ALL the methods. Review only diff from the
previous API.
- run: make remaining-int-methods-filtered.txt
- review remaining-int-methods-filtered.txt. git diff helps.
There are some methods that might have parameters and/or return
values to be converted, as well as properties that come from
non-constant fields.
- Edit methodmap.ext.csv to add new entries if a method needs
metadata fixup.
- Or, edit reduction_rules.txt to add general "rule" to exclude
those mismatches. It is written as a set of replacer regexps
that run on vim.
- Or, edit blacklist-method.txt to add description on "do not
convert to enums" methods, or blacklist-field.txt for fields.
- in general those notes do not describe *everything*. git diff
would tell what is new and what should be taken care this time.
- Here is how I worked on it at API Level 26 to add new lines to methodmap.ext.csv:
- Read git diff on remaining-int-consts.txt to iterate all the new enum types.
- For each new enum type that (typically) has prefix P_ in type T, I run `grep -R P_ | grep T | grep "\w*.html:"` in the local reference docs, to see which documentation HTML mention that const (I ran the command on GNOME terminal, so I got the final grep matches with color highlights).
- And for each HTML doc, look up that const and add records to methodmap.ext.csv.
- I usually bring in mistakes. When there are wrong const mappings, they will result in binding generator warnings for unmatched XPath selectors that you can find them by looking up enummetadata.
- For method mappings, mistakes can be twofolds:
- For the wrong matching, you'll see them as binding generator warnings for unmatched XPath selectors just like const mappings.
- For the wrong replacement enum type, that's NOT going to be unmatched XPath. Instead it will result in a reference to missing type warning that can be still found as a generator warning (but harder to find).
- It is often hard to identify why the mappings are regarded as invalid; sometimes Google API reference tells you lies(!) that there are some methods that actually don't exist(!). You'd like to check `obj/Debug/android-*/api.xml`, `obj/Debug/android-*/api.xml.fixed` (XML after applying metadata fixup) or `Profiles/api-*.xml.in` (for "raw" API data before `api-merge` step).
- At some stage, go to src/Mono.Android and append
methodmap.ext.csv contents to methodmap.csv and run
"make API_LEVELS=[latest] clean all" to get enumified dll.
Then you can iterate this review cycle with reduced
remaining-int-methods-filtered.txt entries.
- Sometimes reduction-rules.txt describes regexps too broad
and it could happen that they incorrectly hides some members
that must be enumified. Ideally every single rule should be
reviewed after finishing this enumification step.
- It is very easy to bring typo, and typos could result in bad
scenario e.g. the method simply disappears because the enum
type *does not exist*. Or it could be *ignored* due to wrong
matching. So you would always want to make sure that those
members exist and enumification successfully done.
gui-compare is your friend to compare old and new assemblies.
* Old files
remaining-int-fields.txt is extraneous and will be removed;
in C#, they become properties and cannot distinguish method-based
getter/setter without implementation.
This was collected from some XPath operations (I don't even remember),
but anyways no need to review them separately.
It is part of remaining-int-methods.txt now.
intermediary-enum-list.txt is not useful at least nowadays.
This is originally meant to be used for reviewing methods automatically
and that methodology is not in use anymore.
Enumifiable int constants could be (relatively) easily reviewed by
remaining-int-consts.txt* files along with blacklist-const.txt.
intermediary-enum-candidates.txt is not in use. Not useful for the
same reason as intermediary-enum-list.txt.
7ebd6bf888