276 строки
13 KiB
Plaintext
276 строки
13 KiB
Plaintext
|
|
This directory contains a couple of "helper" tools for API int-to-enum conversion.
|
|
|
|
This is partly done by scraping Android API reference, but it's mostly done by human.
|
|
|
|
The results are to be saved as "map.ext.csv" and "methodmap.ext.csv" that are to be added to "map.csv" and "methodmap.csv" in Mono.Android source.
|
|
|
|
* "doc scraper" part
|
|
|
|
With this approach, we can colledct possible enum values in generic approach.
|
|
|
|
Pros:
|
|
- It could reduce massive manual edits.
|
|
|
|
Cons:
|
|
- It is not automatically doable to map enum types to methods, and this cannot be
|
|
done without "filtering out" "already-done" methods.
|
|
- There are lots of arbitrarily named enum types that cannot be automatically mapped.
|
|
|
|
Required tools will be described below (some are done, but not limited to them).
|
|
|
|
** Const lookup
|
|
|
|
*** generate-const-list.exe
|
|
|
|
This tool collects "Constants" from the SDK reference pages into XML.
|
|
|
|
The result is printed to stdout, stored as const-list-*.xml by Makefile.
|
|
|
|
** Method argument lookup - not really in practical use
|
|
|
|
This approach has never been used, and won't be in use because it is
|
|
now much easier to review methods and properties using manually-
|
|
maintained list.
|
|
|
|
*** method lookup
|
|
|
|
- for each constants listed in const-list-* XML, grep the constant name as in URL form
|
|
and collect "which documents mentions the value" list. Key is document, value is
|
|
a list of enums. This is done for efficient document parse.
|
|
- For each key doc, load and get relevant methods, then store the relation map into
|
|
another XML, keyed by each method.
|
|
|
|
*** generate-enumlist-to-query.exe
|
|
|
|
This tool generates a flat list text from const-list-*.xml for input to other tool.
|
|
|
|
The result is printed to stdout, stored as intermediary-enum-list.txt by Makefile.
|
|
|
|
*** generate-intermediary-doc-enum-mapping.exe
|
|
|
|
This tool generates "which documents mention the value" mappings.
|
|
|
|
WARNING: this may take many hours to complete. You had better run this tool only when
|
|
a new API Level is out.
|
|
|
|
It takes reference doc and intermediary-enum-list.txt.
|
|
|
|
The result is printed to stdout, stored as intermediary-doc-enum-mapping.tsv by Makefile.
|
|
|
|
*** generate-intermediary-enum-candidates.exe
|
|
|
|
This tool looks up types that have more than one fields that share prefixes, which is,
|
|
substring before '_'. For example, there are DEFAULT_KEY_DIALER and DEFAULT_KEYS_DISABLE
|
|
in android.app.Activity, hence android.app.Activity matches this condition with "DEFAULT_".
|
|
|
|
*** generate-const-mapping.exe
|
|
|
|
This tool converts <enum-conversion-mappings><map> elements into map.csv.
|
|
|
|
The <map> element can be half-automatically generated by generate-intermediary-enum-candidates.exe.
|
|
|
|
TODO: I have added "merge" attribute that indicates "merging" the specified enum values into current enums.
|
|
This is required for implemented interface consts such as ContactsContract$CommonDataKinds$Email
|
|
for EmailDataKind that should be merged with ContactsContract$CommonDataKinds$CommonColumns BaseDataKind.
|
|
|
|
* C# code lookup with blacklisting
|
|
|
|
With this approach, we collect "public const int" fields and examine if they should
|
|
be turned into enums, and "blacklist" those constants that should not become enum.
|
|
|
|
** lookup
|
|
|
|
I have identified some namespaces and types that should not be included in the
|
|
results. Therefore, this lookup can be done as:
|
|
|
|
grep "const int " Mono.Android/platforms/android-14/src/generated/*.cs \
|
|
| grep -v Javax.Microedition \ # opengl
|
|
| grep -v Dalvik.Bytecode \ # dex opcodes
|
|
| grep -v Android.Resource.cs \ # R.* fields that has to be kept as int
|
|
| grep -v BluetoothAssignedNumbers \ # describes several companies, listed just "examples", could be extended or take custom values.
|
|
| grep -v GLES \ # android.opengl
|
|
| grep -v ContentsFileDescriptor \ # Parcelable.describeContents() return, used by many types
|
|
| grep -v ParcelableWriteReturnValue \ # Parcelable, flag without any other member
|
|
|
|
Then we lookup methods that contains int argument or int return value. This can be
|
|
looked up "public", "int", "(" and ")", except those which contain "const" (otherwise
|
|
this search results will contain consts).
|
|
|
|
grep "public" | grep -v "const" | grep "int" | grep "(" | grep ")" \
|
|
(exclusion follows)
|
|
|
|
The results were saved as remaining-int-consts.txt. We stored past
|
|
final resultsper API level (since 15, as the original work started
|
|
as of API Level 14 and versioning came up after that level).
|
|
|
|
** examine consts and methods
|
|
|
|
It is easier to begin with constaints with many values.
|
|
|
|
generate-const-list.exe output will be helpful.
|
|
|
|
** blacklisting
|
|
|
|
First to note: this part is not helpful for automated tasks.
|
|
They are rather historical records and we don't even read them
|
|
nowadays, so they would be helpful only when you are curious
|
|
"why this is not enumified...?"
|
|
|
|
Field blacklist is described in either namespace, class or individual field name.
|
|
Then build input to "uniq" as in follows: grep namespace(,class(,field)) from the
|
|
first grep results.
|
|
|
|
(This is not really *automatically* done in the actual enum conversion
|
|
work; the list is maintained manually.)
|
|
|
|
* verification
|
|
|
|
There is no automatic verification (at least yet).
|
|
Though there will be some verification aids:
|
|
|
|
** reduction_rules.txt
|
|
|
|
"make remaining-int-methods.txt" creates the text that contains
|
|
int and '[(\[]' from the sources (in API LEVEL 16).
|
|
|
|
copy it to some dummy file and open vi.
|
|
Paste reduction_rules.txt contents, which run lots of replaces.
|
|
(could be sed script...)
|
|
grep "int[\[ ]" against the output file and get filtered results.
|
|
|
|
add methodmap.ext.csv entries based on the remaining lines above.
|
|
|
|
** When new API level arrives ...
|
|
|
|
(updated: this description was initially written as of 14 as the
|
|
latest API version, then slightly updated when 15 actually arrived.)
|
|
|
|
Right now remaining-int-methods.txt is the source to apply filter.
|
|
When the new API level arrives, it should be able to create a
|
|
diff between the "remaining" lines between new and old API levels.
|
|
|
|
So, get the diff and use this for filtering input. But beware,
|
|
the filters should be re-examined as those filters are examined
|
|
through human check e.g. "OK, there is no enum-convertible "maxBlah"
|
|
in the remaining methods, I reviewed them".
|
|
|
|
When making changes in mappings, make sure that the API does not
|
|
"vanish" due to wrong mappings. Wrong reduction rule or wrong mapping
|
|
path (package/type/method/parameter) do not harm much (as it will
|
|
just fail to reduce methods to review), but if you have wrong
|
|
destination type, it will just disappear.
|
|
Corcompare significantly helps reducing this kind of mistakes much.
|
|
|
|
For constants and fields, it's easier as we have far less lines
|
|
to check, and we store the remaining lines in API level 14 in git.
|
|
|
|
This is a diff between 14 and 15, for example:
|
|
https://gist.github.com/d3df1c611055e4620d02
|
|
|
|
This is still painful, but not huge as now we have filter candidates
|
|
and we only have to review diffs, not the entire API.
|
|
|
|
*** Tasks to do, step by step
|
|
|
|
- edit Makefile and replace any *previous* version number to the latest number.
|
|
- run make
|
|
- have a look at remaining-int-consts.txt. git diff differentiates
|
|
longstanding ones and new ones. There'd be a lot new at first.
|
|
They have to be reduced to the "we reviewed all of those
|
|
constants" state.
|
|
- They can be reduced by adding more enums. Add more entries on
|
|
enum-conversion-mappings.xml, run make to get map.ext.csv and
|
|
replace additional lines in src/Mono.Android/map.csv with the
|
|
entire map.ext.csv contents.
|
|
(after "// automatically generated ones" line)
|
|
|
|
Note that "// automatically generated ones" is semantically
|
|
important and recognized by enum mapping parser to determine
|
|
whether to generate corresponding [Obsolete] field or not.
|
|
|
|
- for additional entries to good-old constants (which must NOT be
|
|
after "// automatically generated ones" line) just edit map.csv.
|
|
- then run "make API_LEVELS=[latest] clean all" in src/Mono.Android
|
|
to build enumified dll.
|
|
- note tht API Level description in the API docs are used to
|
|
become part of the CSV entry. The first item is the API Level.
|
|
That often brings problem. This is what happened when API Level
|
|
26 as stable has arrived: Google had published "android-26" API
|
|
which used to be "android-O", they still hadn't published the
|
|
corresponding docs and therefore the docs say "O" but the actual
|
|
API was "26". That later brought confusion at generator.exe ran.
|
|
To workaround this issue, replace the wrong API level specifier
|
|
(e.g. "O") with the actual API level (e.g. "26"). Use regex like
|
|
/^O/26/ to achieve this.
|
|
- go back to this directory and rebuild remaining-int-consts.txt
|
|
and check if the enumified lines disappeared.
|
|
- There are some constaints that are NOT enumified
|
|
- Some of the int consts are on blacklist-int-consts.txt.
|
|
This list also describes the reason why they are left as int.
|
|
- No need to enumify @Deprecated constants.
|
|
- There are "argurably enumifiable" constants or "enumifiable
|
|
but brings breakage" constants. They are commented on map.csv
|
|
immediately before "// automatically generated ones" line.
|
|
- Once int constants are converted to enums, the next step is to
|
|
review all the API methods to find what parameters and return
|
|
values need to be changed to those new enums. Note that it is
|
|
impossible to review ALL the methods. Review only diff from the
|
|
previous API.
|
|
- run: make remaining-int-methods-filtered.txt
|
|
- review remaining-int-methods-filtered.txt. git diff helps.
|
|
There are some methods that might have parameters and/or return
|
|
values to be converted, as well as properties that come from
|
|
non-constant fields.
|
|
- Edit methodmap.ext.csv to add new entries if a method needs
|
|
metadata fixup.
|
|
- Or, edit reduction_rules.txt to add general "rule" to exclude
|
|
those mismatches. It is written as a set of replacer regexps
|
|
that run on vim.
|
|
- Or, edit blacklist-method.txt to add description on "do not
|
|
convert to enums" methods, or blacklist-field.txt for fields.
|
|
- in general those notes do not describe *everything*. git diff
|
|
would tell what is new and what should be taken care this time.
|
|
- Here is how I worked on it at API Level 26 to add new lines to methodmap.ext.csv:
|
|
- Read git diff on remaining-int-consts.txt to iterate all the new enum types.
|
|
- For each new enum type that (typically) has prefix P_ in type T, I run `grep -R P_ | grep T | grep "\w*.html:"` in the local reference docs, to see which documentation HTML mention that const (I ran the command on GNOME terminal, so I got the final grep matches with color highlights).
|
|
- And for each HTML doc, look up that const and add records to methodmap.ext.csv.
|
|
- I usually bring in mistakes. When there are wrong const mappings, they will result in binding generator warnings for unmatched XPath selectors that you can find them by looking up enummetadata.
|
|
- For method mappings, mistakes can be twofolds:
|
|
- For the wrong matching, you'll see them as binding generator warnings for unmatched XPath selectors just like const mappings.
|
|
- For the wrong replacement enum type, that's NOT going to be unmatched XPath. Instead it will result in a reference to missing type warning that can be still found as a generator warning (but harder to find).
|
|
- It is often hard to identify why the mappings are regarded as invalid; sometimes Google API reference tells you lies(!) that there are some methods that actually don't exist(!). You'd like to check `obj/Debug/android-*/api.xml`, `obj/Debug/android-*/api.xml.fixed` (XML after applying metadata fixup) or `Profiles/api-*.xml.in` (for "raw" API data before `api-merge` step).
|
|
- At some stage, go to src/Mono.Android and append
|
|
methodmap.ext.csv contents to methodmap.csv and run
|
|
"make API_LEVELS=[latest] clean all" to get enumified dll.
|
|
Then you can iterate this review cycle with reduced
|
|
remaining-int-methods-filtered.txt entries.
|
|
- Sometimes reduction-rules.txt describes regexps too broad
|
|
and it could happen that they incorrectly hides some members
|
|
that must be enumified. Ideally every single rule should be
|
|
reviewed after finishing this enumification step.
|
|
- It is very easy to bring typo, and typos could result in bad
|
|
scenario e.g. the method simply disappears because the enum
|
|
type *does not exist*. Or it could be *ignored* due to wrong
|
|
matching. So you would always want to make sure that those
|
|
members exist and enumification successfully done.
|
|
gui-compare is your friend to compare old and new assemblies.
|
|
|
|
* Old files
|
|
|
|
remaining-int-fields.txt is extraneous and will be removed;
|
|
in C#, they become properties and cannot distinguish method-based
|
|
getter/setter without implementation.
|
|
This was collected from some XPath operations (I don't even remember),
|
|
but anyways no need to review them separately.
|
|
It is part of remaining-int-methods.txt now.
|
|
|
|
intermediary-enum-list.txt is not useful at least nowadays.
|
|
This is originally meant to be used for reviewing methods automatically
|
|
and that methodology is not in use anymore.
|
|
Enumifiable int constants could be (relatively) easily reviewed by
|
|
remaining-int-consts.txt* files along with blacklist-const.txt.
|
|
|
|
intermediary-enum-candidates.txt is not in use. Not useful for the
|
|
same reason as intermediary-enum-list.txt.
|